Jan 3 00:00:06 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client f3a710ae-5df3-01f3-301a-d9c5d3a5821e (at 10.151.50.183@o2ib) in 157 seconds. I think it's dead, and I am evicting it. Jan 3 00:00:06 service103 kernel: Lustre: Skipped 104 previous similar messages Jan 3 00:01:22 service103 kernel: Lustre: nbp6-OST003a: haven't heard from client 720b274e-3d67-13d4-cebe-924ba307302f (at 10.151.50.184@o2ib) in 225 seconds. I think it's dead, and I am evicting it. Jan 3 00:01:22 service103 kernel: Lustre: Skipped 59 previous similar messages Jan 3 00:01:37 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.51.0@o2ib [old ver: 12, new ver: 12] Jan 3 00:01:37 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 00:02:38 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 070534d6-ebb6-9848-9d4a-29a5af979a46 (at 10.151.51.0@o2ib) in 187 seconds. I think it's dead, and I am evicting it. Jan 3 00:02:38 service103 kernel: Lustre: Skipped 149 previous similar messages Jan 3 00:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 00:11:30 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.21@o2ib [old ver: 12, new ver: 12] Jan 3 00:11:30 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 00:13:38 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 007c73f7-8e9f-29e7-79cb-3469733a8c29 (at 10.151.48.21@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 00:13:38 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 01:09:58 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.47@o2ib [old ver: 12, new ver: 12] Jan 3 01:10:04 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.57@o2ib [old ver: 12, new ver: 12] Jan 3 01:10:04 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 7 previous similar messages Jan 3 01:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 01:12:03 service103 kernel: Lustre: nbp6-OST0022: haven't heard from client a4ce30a1-ad77-c250-86b3-fee70a61dd00 (at 10.151.50.43@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 01:12:03 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 01:12:46 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.165@o2ib [old ver: 12, new ver: 12] Jan 3 01:13:19 service103 kernel: Lustre: nbp6-OST0042: haven't heard from client 664832ac-b26b-71ac-513b-e652df570ca4 (at 10.151.50.77@o2ib) in 168 seconds. I think it's dead, and I am evicting it. Jan 3 01:13:19 service103 kernel: Lustre: Skipped 134 previous similar messages Jan 3 01:14:35 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 83dea352-a72e-7d2f-ec2b-cd755083c935 (at 10.151.50.76@o2ib) in 226 seconds. I think it's dead, and I am evicting it. Jan 3 01:14:35 service103 kernel: Lustre: Skipped 134 previous similar messages Jan 3 01:14:46 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.51.59@o2ib [old ver: 12, new ver: 12] Jan 3 01:14:46 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 10 previous similar messages Jan 3 01:15:51 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client a43dea76-0254-c53f-6497-abbe862cbce6 (at 10.151.51.59@o2ib) in 178 seconds. I think it's dead, and I am evicting it. Jan 3 01:15:51 service103 kernel: Lustre: Skipped 89 previous similar messages Jan 3 01:50:39 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.37@o2ib [old ver: 12, new ver: 12] Jan 3 01:51:48 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.21@o2ib [old ver: 12, new ver: 12] Jan 3 01:51:48 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 6 previous similar messages Jan 3 01:52:48 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client efaa2b2f-d2bb-2f8f-06ce-b9cd2ce04dc7 (at 10.151.48.46@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 01:52:48 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 01:54:49 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.22@o2ib [old ver: 12, new ver: 12] Jan 3 01:54:49 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 01:55:35 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client e739436f-338f-914e-95c8-2afddd42643f (at 10.151.50.139@o2ib) in 188 seconds. I think it's dead, and I am evicting it. Jan 3 01:55:35 service103 kernel: Lustre: Skipped 179 previous similar messages Jan 3 02:02:21 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.182@o2ib [old ver: 12, new ver: 12] Jan 3 02:02:21 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 02:03:19 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.173@o2ib [old ver: 12, new ver: 12] Jan 3 02:03:19 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 02:04:28 service103 kernel: Lustre: nbp6-OST0072: haven't heard from client c3f752cb-6e44-2b7e-f6d8-a013d98f1662 (at 10.151.51.0@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 02:04:28 service103 kernel: Lustre: Skipped 119 previous similar messages Jan 3 02:05:44 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 8337300e-f3b1-29ed-78c5-699e2214bd0e (at 10.151.50.38@o2ib) in 183 seconds. I think it's dead, and I am evicting it. Jan 3 02:05:44 service103 kernel: Lustre: Skipped 74 previous similar messages Jan 3 02:07:11 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.177@o2ib [old ver: 12, new ver: 12] Jan 3 02:08:37 service103 kernel: Lustre: nbp6-OST0052: haven't heard from client 48b41b67-43a4-6f89-12d1-9c1011de8397 (at 10.151.50.164@o2ib) in 189 seconds. I think it's dead, and I am evicting it. Jan 3 02:08:37 service103 kernel: Lustre: Skipped 119 previous similar messages Jan 3 02:09:53 service103 kernel: Lustre: 6604:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: 3288ff8d-4158-6672-f2ac-e4af1eef7f84 reconnecting Jan 3 02:09:53 service103 kernel: Lustre: 6604:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Jan 3 02:09:53 service103 kernel: Lustre: 17949:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST003a: 3fd0ba25-c28a-65ec-dfa1-58a9c3b7dbe9 reconnecting Jan 3 02:09:53 service103 kernel: Lustre: 17949:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 12 previous similar messages Jan 3 02:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 02:11:05 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.162@o2ib [old ver: 12, new ver: 12] Jan 3 02:11:05 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 5 previous similar messages Jan 3 02:13:10 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client a1a6a2bd-09f5-3930-c4df-b619d1bc43aa (at 10.151.50.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 02:13:10 service103 kernel: Lustre: Skipped 89 previous similar messages Jan 3 02:14:26 service103 kernel: Lustre: nbp6-OST0012: haven't heard from client a69a7b92-3ef5-2d0d-e28f-5ada59f7b36c (at 10.151.51.0@o2ib) in 205 seconds. I think it's dead, and I am evicting it. Jan 3 02:14:26 service103 kernel: Lustre: Skipped 224 previous similar messages Jan 3 02:16:17 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.51.59@o2ib [old ver: 12, new ver: 12] Jan 3 02:16:17 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 22 previous similar messages Jan 3 02:18:11 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 3fd0ba25-c28a-65ec-dfa1-58a9c3b7dbe9 (at 10.151.50.165@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 02:18:11 service103 kernel: Lustre: Skipped 119 previous similar messages Jan 3 03:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 04:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 05:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 06:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 07:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 08:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 08:18:33 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.30.7@o2ib [old ver: 12, new ver: 12] Jan 3 08:18:33 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 08:20:13 service103 kernel: Lustre: nbp6-OST001a: haven't heard from client c22729b0-f8db-a25e-3dc7-7c57860fd59b (at 10.151.30.6@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 08:20:13 service103 kernel: Lustre: Skipped 59 previous similar messages Jan 3 09:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 09:27:02 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 536815bd-dc8f-06fa-84f5-d072c6f3d38b (at 10.151.0.201@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 09:27:02 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 09:32:25 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 531f51cb-a0aa-0d1e-9f0f-c7de01cfb755 (at 10.151.58.181@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 09:32:25 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 09:43:34 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 03309993-3f0e-bf55-854b-a4bbc4ebedb8 (at 10.151.30.11@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 09:43:34 service103 kernel: Lustre: Skipped 599 previous similar messages Jan 3 09:50:11 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 4db25b79-06c2-901c-6c0b-7d32cf13ce8f (at 10.151.30.61@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 09:50:11 service103 kernel: Lustre: Skipped 959 previous similar messages Jan 3 10:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 10:32:48 service103 kernel: Lustre: 3233:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.10.167@o2ib [old ver: 12, new ver: 12] Jan 3 10:35:02 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client af02a355-4790-e753-8c4c-88f50dd4b97f (at 10.151.10.167@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 10:35:02 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 11:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 11:14:58 service103 kernel: host3: ib_srp: DREQ received - connection closed Jan 3 11:15:00 service103 kernel: sd 2:0:0:27: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:15:00 service103 kernel: host3: ib_srp: connection closed Jan 3 11:15:13 service103 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Jan 3 11:15:18 service103 OpenSM[4724]: SM port is down Jan 3 11:15:18 service103 OpenSM[4724]: Entering DISCOVERING state Jan 3 11:15:28 service103 OpenSM[4724]: SM port is down Jan 3 11:15:48 service103 last message repeated 2 times Jan 3 11:15:49 service103 kernel: host3: ib_srp: failed send status 12 Jan 3 11:15:49 service103 kernel: host3: ib_srp: failed receive status 5 Jan 3 11:15:49 service103 kernel: host3: ib_srp: failed receive status 5 Jan 3 11:15:58 service103 OpenSM[4724]: SM port is down Jan 3 11:16:08 service103 OpenSM[4724]: SM port is down Jan 3 11:16:09 service103 run_srp_daemon[22277]: failed srp_daemon: [HCA=mlx4_0] [port=1] [exit status=110]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 11:16:13 service103 multipathd: sdc: tur checker reports path is down Jan 3 11:16:13 service103 multipathd: checker failed path 8:32 in map ddn6a-nbp6-ost2 Jan 3 11:16:13 service103 kernel: device-mapper: multipath: Failing path 8:32. Jan 3 11:16:13 service103 multipathd: ddn6a-nbp6-ost2: remaining active paths: 1 Jan 3 11:16:14 service103 kernel: host3: ib_srp: Got failed path rec status -110 Jan 3 11:16:14 service103 kernel: host3: ib_srp: Path record query failed Jan 3 11:16:14 service103 kernel: host3: ib_srp: reconnect failed (-110), removing target port. Jan 3 11:16:14 service103 multipathd: sdd: tur checker reports path is down Jan 3 11:16:14 service103 multipathd: checker failed path 8:48 in map ddn6a-nbp6-ost10 Jan 3 11:16:14 service103 kernel: device-mapper: multipath: Failing path 8:48. Jan 3 11:16:14 service103 multipathd: ddn6a-nbp6-ost10: remaining active paths: 1 Jan 3 11:16:14 service103 kernel: scsi 3:0:0:19: rejecting I/O to dead device Jan 3 11:16:14 service103 multipathd: sdf: tur checker reports path is down Jan 3 11:16:14 service103 multipathd: checker failed path 8:80 in map ddn6a-nbp6-ost18 Jan 3 11:16:14 service103 kernel: device-mapper: multipath: Failing path 8:80. Jan 3 11:16:14 service103 multipathd: ddn6a-nbp6-ost18: remaining active paths: 1 Jan 3 11:16:14 service103 kernel: scsi 3:0:0:27: rejecting I/O to dead device Jan 3 11:16:14 service103 multipathd: sdg: tur checker reports path is down Jan 3 11:16:14 service103 kernel: device-mapper: multipath: Failing path 8:96. Jan 3 11:16:14 service103 multipathd: checker failed path 8:96 in map ddn6a-nbp6-ost26 Jan 3 11:16:14 service103 kernel: scsi 3:0:0:35: rejecting I/O to dead device Jan 3 11:16:14 service103 multipathd: ddn6a-nbp6-ost26: remaining active paths: 1 Jan 3 11:16:14 service103 kernel: device-mapper: multipath: Failing path 8:112. Jan 3 11:16:14 service103 multipathd: sdh: tur checker reports path is down Jan 3 11:16:14 service103 kernel: scsi 3:0:0:43: rejecting I/O to dead device Jan 3 11:16:15 service103 multipathd: checker failed path 8:112 in map ddn6a-nbp6-ost34 Jan 3 11:16:15 service103 kernel: device-mapper: multipath: Failing path 8:128. Jan 3 11:16:15 service103 multipathd: ddn6a-nbp6-ost34: remaining active paths: 1 Jan 3 11:16:15 service103 kernel: scsi 3:0:0:51: rejecting I/O to dead device Jan 3 11:16:15 service103 multipathd: sdi: tur checker reports path is down Jan 3 11:16:15 service103 kernel: device-mapper: multipath: Failing path 8:144. Jan 3 11:16:15 service103 multipathd: checker failed path 8:128 in map ddn6a-nbp6-ost42 Jan 3 11:16:15 service103 kernel: scsi 3:0:0:59: rejecting I/O to dead device Jan 3 11:16:15 service103 multipathd: ddn6a-nbp6-ost42: remaining active paths: 1 Jan 3 11:16:15 service103 kernel: device-mapper: multipath: Failing path 8:160. Jan 3 11:16:15 service103 multipathd: sdj: tur checker reports path is down Jan 3 11:16:15 service103 kernel: scsi 3:0:0:67: rejecting I/O to dead device Jan 3 11:16:15 service103 multipathd: checker failed path 8:144 in map ddn6a-nbp6-ost50 Jan 3 11:16:15 service103 kernel: device-mapper: multipath: Failing path 8:176. Jan 3 11:16:15 service103 multipathd: ddn6a-nbp6-ost50: remaining active paths: 1 Jan 3 11:16:15 service103 kernel: scsi 3:0:0:75: rejecting I/O to dead device Jan 3 11:16:15 service103 multipathd: sdk: tur checker reports path is down Jan 3 11:16:15 service103 kernel: device-mapper: multipath: Failing path 8:192. Jan 3 11:16:15 service103 multipathd: checker failed path 8:160 in map ddn6a-nbp6-ost58 Jan 3 11:16:15 service103 kernel: scsi 3:0:0:83: rejecting I/O to dead device Jan 3 11:16:16 service103 multipathd: ddn6a-nbp6-ost58: remaining active paths: 1 Jan 3 11:16:16 service103 kernel: device-mapper: multipath: Failing path 8:208. Jan 3 11:16:16 service103 multipathd: sdl: tur checker reports path is down Jan 3 11:16:16 service103 kernel: scsi 3:0:0:91: rejecting I/O to dead device Jan 3 11:16:16 service103 multipathd: checker failed path 8:176 in map ddn6a-nbp6-ost66 Jan 3 11:16:16 service103 kernel: device-mapper: multipath: Failing path 8:224. Jan 3 11:16:16 service103 multipathd: ddn6a-nbp6-ost66: remaining active paths: 1 Jan 3 11:16:16 service103 kernel: scsi 3:0:0:99: rejecting I/O to dead device Jan 3 11:16:16 service103 multipathd: sdm: tur checker reports path is down Jan 3 11:16:16 service103 kernel: device-mapper: multipath: Failing path 8:240. Jan 3 11:16:16 service103 multipathd: checker failed path 8:192 in map ddn6a-nbp6-ost74 Jan 3 11:16:16 service103 kernel: scsi 3:0:0:107: rejecting I/O to dead device Jan 3 11:16:16 service103 multipathd: ddn6a-nbp6-ost74: remaining active paths: 1 Jan 3 11:16:16 service103 kernel: device-mapper: multipath: Failing path 65:0. Jan 3 11:16:16 service103 multipathd: sdn: tur checker reports path is down Jan 3 11:16:16 service103 kernel: scsi 3:0:0:115: rejecting I/O to dead device Jan 3 11:16:16 service103 multipathd: checker failed path 8:208 in map ddn6a-nbp6-ost82 Jan 3 11:16:16 service103 kernel: device-mapper: multipath: Failing path 65:16. Jan 3 11:16:16 service103 multipathd: ddn6a-nbp6-ost82: remaining active paths: 1 Jan 3 11:16:17 service103 multipathd: sdo: tur checker reports path is down Jan 3 11:16:17 service103 multipathd: checker failed path 8:224 in map ddn6a-nbp6-ost90 Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost90: remaining active paths: 1 Jan 3 11:16:17 service103 multipathd: sdp: tur checker reports path is down Jan 3 11:16:17 service103 multipathd: checker failed path 8:240 in map ddn6a-nbp6-ost98 Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost98: remaining active paths: 1 Jan 3 11:16:17 service103 multipathd: sdq: tur checker reports path is down Jan 3 11:16:17 service103 multipathd: checker failed path 65:0 in map ddn6a-nbp6-ost106 Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost106: remaining active paths: 1 Jan 3 11:16:17 service103 multipathd: sdr: tur checker reports path is down Jan 3 11:16:17 service103 multipathd: checker failed path 65:16 in map ddn6a-nbp6-ost114 Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost114: remaining active paths: 1 Jan 3 11:16:17 service103 multipathd: dm-5: add map (uevent) Jan 3 11:16:17 service103 multipathd: dm-5: devmap already registered Jan 3 11:16:17 service103 multipathd: sdc: remove path (uevent) Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:16 10] Jan 3 11:16:17 service103 multipathd: sdd: remove path (uevent) Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:64 10] Jan 3 11:16:17 service103 multipathd: sdf: remove path (uevent) Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:32 10] Jan 3 11:16:17 service103 multipathd: sdg: remove path (uevent) Jan 3 11:16:17 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:48 10] Jan 3 11:16:18 service103 multipathd: sdh: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:64 10] Jan 3 11:16:18 service103 multipathd: dm-6: add map (uevent) Jan 3 11:16:18 service103 multipathd: dm-6: devmap already registered Jan 3 11:16:18 service103 multipathd: sdi: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:80 10] Jan 3 11:16:18 service103 multipathd: sdj: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:96 10] Jan 3 11:16:18 service103 multipathd: sdk: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:112 10] Jan 3 11:16:18 service103 multipathd: sdl: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:128 10] Jan 3 11:16:18 service103 multipathd: sdm: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:144 10] Jan 3 11:16:18 service103 multipathd: sdn: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:160 10] Jan 3 11:16:18 service103 multipathd: sdo: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:176 10] Jan 3 11:16:18 service103 multipathd: sdp: remove path (uevent) Jan 3 11:16:18 service103 OpenSM[4724]: SM port is down Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:192 10] Jan 3 11:16:18 service103 multipathd: sdq: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:208 10] Jan 3 11:16:18 service103 multipathd: sdr: remove path (uevent) Jan 3 11:16:18 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:224 10] Jan 3 11:16:18 service103 multipathd: dm-7: add map (uevent) Jan 3 11:16:18 service103 multipathd: dm-7: devmap already registered Jan 3 11:16:18 service103 multipathd: dm-8: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-8: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-9: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-9: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-10: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-10: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-11: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-11: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-12: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-12: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-13: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-13: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-14: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-14: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-0: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-0: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-1: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-1: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-2: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-2: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-3: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-3: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-4: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-4: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-5: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-5: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-6: add map (uevent) Jan 3 11:16:19 service103 multipathd: dm-6: devmap already registered Jan 3 11:16:19 service103 multipathd: dm-7: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-7: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-8: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-8: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-9: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-9: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-10: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-10: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-11: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-11: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-12: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-12: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-13: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-13: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-14: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-14: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-0: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-0: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-1: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-1: devmap already registered Jan 3 11:16:20 service103 multipathd: dm-2: add map (uevent) Jan 3 11:16:20 service103 multipathd: dm-2: devmap already registered Jan 3 11:16:25 service103 run_srp_daemon[22836]: starting srp_daemon: [HCA=mlx4_0] [port=1] Jan 3 11:16:28 service103 OpenSM[4724]: SM port is down Jan 3 11:16:48 service103 last message repeated 2 times Jan 3 11:16:58 service103 OpenSM[4724]: Entering MASTER state Jan 3 11:16:58 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jan 3 11:16:58 service103 OpenSM[4724]: SUBNET UP Jan 3 11:16:58 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Jan 3 11:20:15 service103 kernel: sd 2:0:0:83: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:160 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:176 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:192 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:208 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:224 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:16 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:64 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:32 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:48 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:64 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:80 10] Jan 3 11:20:15 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:96 10] Jan 3 11:20:21 service103 kernel: sd 2:0:0:35: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:21 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:21 service103 kernel: Jan 3 11:20:21 service103 kernel: end_request: I/O error, dev sdu, sector 13442713424 Jan 3 11:20:21 service103 kernel: device-mapper: multipath: Failing path 65:64. Jan 3 11:20:21 service103 kernel: sd 2:0:0:35: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:21 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:21 service103 kernel: Jan 3 11:20:21 service103 kernel: end_request: I/O error, dev sdu, sector 7516996680 Jan 3 11:20:21 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:112 10] Jan 3 11:20:21 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:128 10] Jan 3 11:20:21 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:144 10] Jan 3 11:20:21 service103 multipathd: dm-0: add map (uevent) Jan 3 11:20:21 service103 multipathd: dm-0: devmap already registered Jan 3 11:20:21 service103 kernel: sd 2:0:0:107: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:21 service103 multipathd: dm-1: add map (uevent) Jan 3 11:20:21 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:22 service103 multipathd: dm-1: devmap already registered Jan 3 11:20:22 service103 kernel: Jan 3 11:20:22 service103 multipathd: dm-2: add map (uevent) Jan 3 11:20:22 service103 kernel: end_request: I/O error, dev sdad, sector 8945014624 Jan 3 11:20:22 service103 multipathd: dm-2: devmap already registered Jan 3 11:20:22 service103 kernel: device-mapper: multipath: Failing path 65:208. Jan 3 11:20:22 service103 multipathd: dm-3: add map (uevent) Jan 3 11:20:22 service103 kernel: sd 2:0:0:107: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:22 service103 multipathd: dm-3: devmap already registered Jan 3 11:20:22 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:22 service103 multipathd: dm-4: add map (uevent) Jan 3 11:20:22 service103 kernel: Jan 3 11:20:22 service103 multipathd: dm-4: devmap already registered Jan 3 11:20:22 service103 kernel: end_request: I/O error, dev sdad, sector 7516480336 Jan 3 11:20:22 service103 multipathd: dm-5: add map (uevent) Jan 3 11:20:22 service103 kernel: sd 2:0:0:59: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:22 service103 multipathd: dm-5: devmap already registered Jan 3 11:20:22 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:22 service103 multipathd: dm-6: add map (uevent) Jan 3 11:20:23 service103 kernel: Jan 3 11:20:23 service103 multipathd: dm-6: devmap already registered Jan 3 11:20:23 service103 kernel: end_request: I/O error, dev sdx, sector 14317948128 Jan 3 11:20:23 service103 multipathd: dm-7: add map (uevent) Jan 3 11:20:23 service103 kernel: device-mapper: multipath: Failing path 65:112. Jan 3 11:20:23 service103 multipathd: dm-7: devmap already registered Jan 3 11:20:23 service103 kernel: sd 2:0:0:59: Device not ready: <6>: Current: sense key: Not Ready Jan 3 11:20:23 service103 multipathd: dm-8: add map (uevent) Jan 3 11:20:23 service103 kernel: Add. Sense: Logical unit not accessible, target port in unavailable state Jan 3 11:20:23 service103 multipathd: dm-8: devmap already registered Jan 3 11:20:23 service103 kernel: Jan 3 11:20:23 service103 multipathd: dm-9: add map (uevent) Jan 3 11:20:23 service103 kernel: end_request: I/O error, dev sdx, sector 7516601656 Jan 3 11:20:23 service103 multipathd: dm-9: devmap already registered Jan 3 11:20:23 service103 kernel: host2: ib_srp: DREQ received - connection closed Jan 3 11:20:23 service103 multipathd: dm-10: add map (uevent) Jan 3 11:20:23 service103 multipathd: dm-10: devmap already registered Jan 3 11:20:24 service103 multipathd: dm-11: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-11: devmap already registered Jan 3 11:20:24 service103 multipathd: dm-12: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-12: devmap already registered Jan 3 11:20:24 service103 multipathd: dm-13: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-13: devmap already registered Jan 3 11:20:24 service103 multipathd: dm-14: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-14: devmap already registered Jan 3 11:20:24 service103 multipathd: dm-9: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-9: devmap already registered Jan 3 11:20:24 service103 multipathd: 65:64: mark as failed Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost34: Entering recovery mode: max_retries=12 Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost34: remaining active paths: 0 Jan 3 11:20:24 service103 multipathd: dm-3: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-3: devmap already registered Jan 3 11:20:24 service103 multipathd: 65:208: mark as failed Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost106: Entering recovery mode: max_retries=12 Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost106: remaining active paths: 0 Jan 3 11:20:24 service103 multipathd: dm-12: add map (uevent) Jan 3 11:20:24 service103 multipathd: dm-12: devmap already registered Jan 3 11:20:24 service103 multipathd: 65:112: mark as failed Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost58: Entering recovery mode: max_retries=12 Jan 3 11:20:24 service103 kernel: host2: ib_srp: connection closed Jan 3 11:20:24 service103 multipathd: ddn6a-nbp6-ost58: remaining active paths: 0 Jan 3 11:20:37 service103 kernel: ib_srp: ASYNC event= 10 on device= mlx4_1 Jan 3 11:20:44 service103 OpenSM[4756]: SM port is down Jan 3 11:20:44 service103 OpenSM[4756]: Entering DISCOVERING state Jan 3 11:20:54 service103 OpenSM[4756]: SM port is down Jan 3 11:20:58 service103 kernel: host2: ib_srp: failed send status 12 Jan 3 11:20:58 service103 kernel: host2: ib_srp: failed send status 5 Jan 3 11:20:58 service103 kernel: host2: ib_srp: failed send status 5 Jan 3 11:21:04 service103 OpenSM[4756]: SM port is down Jan 3 11:21:11 service103 kernel: scsi4 : SRP.T10:1A6D0F0003C90200 Jan 3 11:21:11 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:11 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 11:21:11 service103 kernel: scsi 4:0:0:0: Attached scsi generic sg5 type 12 Jan 3 11:21:11 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:11 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:11 service103 kernel: sd 4:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:21:11 service103 kernel: sdc: Unit Not Ready, sense: Jan 3 11:21:11 service103 kernel: : Current: sense key: Unit Attention Jan 3 11:21:11 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 11:21:11 service103 kernel: Jan 3 11:21:11 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 11:21:11 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:11 service103 kernel: sdc: Write Protect is off Jan 3 11:21:11 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 11:21:11 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 11:21:11 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:11 service103 kernel: sdc: Write Protect is off Jan 3 11:21:11 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 11:21:11 service103 kernel: sdc: unknown partition table Jan 3 11:21:11 service103 kernel: sd 4:0:0:3: Attached scsi disk sdc Jan 3 11:21:11 service103 kernel: sd 4:0:0:3: Attached scsi generic sg6 type 0 Jan 3 11:21:11 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:12 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:12 service103 logger: Adjusted blockdev Jan 3 11:21:12 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 11:21:12 service103 logger: Adjusted blockdev Jan 3 11:21:12 service103 logger: Adjusted sdd max_sectors_kb=4096 Jan 3 11:21:12 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:12 service103 logger: Adjusted sdc max_sectors_kb=4096 Jan 3 11:21:12 service103 logger: Adjusted blockdev Jan 3 11:21:12 service103 logger: Adjusted sdd scheduler=deadline Jan 3 11:21:12 service103 kernel: sdd: Write Protect is off Jan 3 11:21:12 service103 logger: Adjusted sdc scheduler=deadline Jan 3 11:21:12 service103 logger: Adjusted sdf max_sectors_kb=4096 Jan 3 11:21:12 service103 logger: Adjected sdd timeout=280 Jan 3 11:21:12 service103 logger: Adjusted blockdev Jan 3 11:21:12 service103 logger: Adjected sdc timeout=280 Jan 3 11:21:12 service103 logger: Adjusted sdf scheduler=deadline Jan 3 11:21:12 service103 logger: Adjusted blockdev Jan 3 11:21:13 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 11:21:13 service103 logger: Adjusted sdg max_sectors_kb=4096 Jan 3 11:21:13 service103 logger: Adjected sdf timeout=280 Jan 3 11:21:13 service103 logger: Adjusted blockdev Jan 3 11:21:13 service103 logger: Adjusted sdh max_sectors_kb=4096 Jan 3 11:21:13 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 11:21:13 service103 logger: Adjusted sdg scheduler=deadline Jan 3 11:21:13 service103 logger: Adjusted sdi max_sectors_kb=4096 Jan 3 11:21:13 service103 logger: Adjusted blockdev Jan 3 11:21:13 service103 logger: Adjusted sdh scheduler=deadline Jan 3 11:21:13 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:13 service103 logger: Adjected sdg timeout=280 Jan 3 11:21:13 service103 logger: Adjusted sdi scheduler=deadline Jan 3 11:21:13 service103 logger: Adjusted sdj max_sectors_kb=4096 Jan 3 11:21:13 service103 logger: Adjected sdh timeout=280 Jan 3 11:21:13 service103 logger: Adjusted blockdev Jan 3 11:21:13 service103 kernel: sdd: Write Protect is off Jan 3 11:21:13 service103 logger: Adjected sdi timeout=280 Jan 3 11:21:13 service103 logger: Adjusted sdj scheduler=deadline Jan 3 11:21:13 service103 logger: Adjusted sdk max_sectors_kb=4096 Jan 3 11:21:13 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjected sdj timeout=280 Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjusted sdk scheduler=deadline Jan 3 11:21:14 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 11:21:14 service103 logger: Adjusted sdl max_sectors_kb=4096 Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjusted sdm max_sectors_kb=4096 Jan 3 11:21:14 service103 logger: Adjected sdk timeout=280 Jan 3 11:21:14 service103 kernel: sdd: unknown partition table Jan 3 11:21:14 service103 logger: Adjusted sdl scheduler=deadline Jan 3 11:21:14 service103 logger: Adjusted sdn max_sectors_kb=4096 Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjusted sdm scheduler=deadline Jan 3 11:21:14 service103 kernel: sd 4:0:0:11: Attached scsi disk sdd Jan 3 11:21:14 service103 OpenSM[4756]: SM port is down Jan 3 11:21:14 service103 logger: Adjected sdl timeout=280 Jan 3 11:21:14 service103 logger: Adjusted sdn scheduler=deadline Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjusted sdo max_sectors_kb=4096 Jan 3 11:21:14 service103 logger: Adjected sdm timeout=280 Jan 3 11:21:14 service103 kernel: sd 4:0:0:11: Attached scsi generic sg8 type 0 Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:14 service103 logger: Adjected sdn timeout=280 Jan 3 11:21:14 service103 logger: Adjusted sdp max_sectors_kb=4096 Jan 3 11:21:14 service103 logger: Adjusted sdo scheduler=deadline Jan 3 11:21:14 service103 logger: Adjusted blockdev Jan 3 11:21:15 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:15 service103 logger: Adjusted sdq max_sectors_kb=4096 Jan 3 11:21:15 service103 logger: Adjusted sdp scheduler=deadline Jan 3 11:21:15 service103 logger: Adjected sdo timeout=280 Jan 3 11:21:15 service103 logger: Adjusted sdr max_sectors_kb=4096 Jan 3 11:21:15 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:15 service103 logger: Adjusted sdq scheduler=deadline Jan 3 11:21:15 service103 logger: Adjected sdp timeout=280 Jan 3 11:21:15 service103 logger: Adjusted sdr scheduler=deadline Jan 3 11:21:15 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 11:21:15 service103 logger: Adjected sdq timeout=280 Jan 3 11:21:15 service103 logger: Adjected sdr timeout=280 Jan 3 11:21:15 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:15 service103 kernel: sdf: Write Protect is off Jan 3 11:21:15 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 11:21:15 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 11:21:15 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:15 service103 kernel: sdf: Write Protect is off Jan 3 11:21:16 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 11:21:16 service103 kernel: sdf: unknown partition table Jan 3 11:21:16 service103 kernel: sd 4:0:0:19: Attached scsi disk sdf Jan 3 11:21:16 service103 kernel: sd 4:0:0:19: Attached scsi generic sg10 type 0 Jan 3 11:21:16 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:16 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:16 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 11:21:16 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:16 service103 kernel: sdg: Write Protect is off Jan 3 11:21:16 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 11:21:16 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 11:21:16 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:16 service103 kernel: sdg: Write Protect is off Jan 3 11:21:16 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 11:21:17 service103 kernel: sdg: unknown partition table Jan 3 11:21:17 service103 kernel: sd 4:0:0:27: Attached scsi disk sdg Jan 3 11:21:17 service103 kernel: sd 4:0:0:27: Attached scsi generic sg11 type 0 Jan 3 11:21:17 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:17 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:17 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 11:21:17 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:17 service103 kernel: sdh: Write Protect is off Jan 3 11:21:17 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 11:21:17 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 11:21:17 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:17 service103 kernel: sdh: Write Protect is off Jan 3 11:21:18 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 11:21:18 service103 kernel: sdh: unknown partition table Jan 3 11:21:18 service103 kernel: sd 4:0:0:35: Attached scsi disk sdh Jan 3 11:21:18 service103 kernel: sd 4:0:0:35: Attached scsi generic sg12 type 0 Jan 3 11:21:18 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:18 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:18 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 11:21:18 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:18 service103 kernel: sdi: Write Protect is off Jan 3 11:21:18 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 11:21:18 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 11:21:18 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:18 service103 kernel: sdi: Write Protect is off Jan 3 11:21:19 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 11:21:19 service103 kernel: sdi: unknown partition table Jan 3 11:21:19 service103 kernel: sd 4:0:0:43: Attached scsi disk sdi Jan 3 11:21:19 service103 kernel: sd 4:0:0:43: Attached scsi generic sg13 type 0 Jan 3 11:21:19 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:19 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:19 service103 kernel: sdj : very big device. try to use READ CAPACITY(16). Jan 3 11:21:19 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:19 service103 kernel: sdj: Write Protect is off Jan 3 11:21:19 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 11:21:19 service103 kernel: sdj : very big device. try to use READ CAPACITY(16). Jan 3 11:21:19 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:19 service103 kernel: sdj: Write Protect is off Jan 3 11:21:20 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 11:21:20 service103 kernel: sdj: unknown partition table Jan 3 11:21:20 service103 kernel: sd 4:0:0:51: Attached scsi disk sdj Jan 3 11:21:20 service103 kernel: sd 4:0:0:51: Attached scsi generic sg14 type 0 Jan 3 11:21:20 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:20 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:20 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 11:21:20 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:20 service103 kernel: sdk: Write Protect is off Jan 3 11:21:21 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 11:21:21 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 11:21:21 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:21 service103 kernel: sdk: Write Protect is off Jan 3 11:21:21 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 11:21:21 service103 kernel: sdk: unknown partition table Jan 3 11:21:21 service103 kernel: sd 4:0:0:59: Attached scsi disk sdk Jan 3 11:21:21 service103 kernel: sd 4:0:0:59: Attached scsi generic sg15 type 0 Jan 3 11:21:21 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:21 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:21 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 11:21:21 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:21 service103 kernel: sdl: Write Protect is off Jan 3 11:21:22 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 11:21:22 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 11:21:22 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:22 service103 kernel: sdl: Write Protect is off Jan 3 11:21:22 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 11:21:22 service103 kernel: sdl: unknown partition table Jan 3 11:21:22 service103 kernel: sd 4:0:0:67: Attached scsi disk sdl Jan 3 11:21:22 service103 kernel: sd 4:0:0:67: Attached scsi generic sg16 type 0 Jan 3 11:21:22 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:22 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:22 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 11:21:22 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:22 service103 kernel: sdm: Write Protect is off Jan 3 11:21:23 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 11:21:23 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 11:21:23 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:23 service103 kernel: sdm: Write Protect is off Jan 3 11:21:23 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 11:21:23 service103 kernel: sdm: unknown partition table Jan 3 11:21:24 service103 kernel: sd 4:0:0:75: Attached scsi disk sdm Jan 3 11:21:24 service103 kernel: sd 4:0:0:75: Attached scsi generic sg17 type 0 Jan 3 11:21:24 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:24 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:24 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 11:21:24 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:24 service103 kernel: sdn: Write Protect is off Jan 3 11:21:24 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 11:21:24 service103 OpenSM[4756]: SM port is down Jan 3 11:21:24 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 11:21:24 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:24 service103 kernel: sdn: Write Protect is off Jan 3 11:21:24 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 11:21:25 service103 kernel: sdn: unknown partition table Jan 3 11:21:25 service103 kernel: sd 4:0:0:83: Attached scsi disk sdn Jan 3 11:21:25 service103 kernel: sd 4:0:0:83: Attached scsi generic sg18 type 0 Jan 3 11:21:25 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:25 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:25 service103 kernel: sdo : very big device. try to use READ CAPACITY(16). Jan 3 11:21:25 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:25 service103 kernel: sdo: Write Protect is off Jan 3 11:21:25 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 11:21:25 service103 kernel: sdo : very big device. try to use READ CAPACITY(16). Jan 3 11:21:25 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:25 service103 kernel: sdo: Write Protect is off Jan 3 11:21:25 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 11:21:25 service103 kernel: sdo: unknown partition table Jan 3 11:21:26 service103 kernel: sd 4:0:0:91: Attached scsi disk sdo Jan 3 11:21:26 service103 kernel: sd 4:0:0:91: Attached scsi generic sg19 type 0 Jan 3 11:21:26 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:26 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:26 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 11:21:26 service103 kernel: SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:26 service103 kernel: sdp: Write Protect is off Jan 3 11:21:26 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 11:21:26 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 11:21:26 service103 kernel: SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:26 service103 kernel: sdp: Write Protect is off Jan 3 11:21:26 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 11:21:26 service103 kernel: sdp: unknown partition table Jan 3 11:21:27 service103 kernel: sd 4:0:0:99: Attached scsi disk sdp Jan 3 11:21:27 service103 kernel: sd 4:0:0:99: Attached scsi generic sg20 type 0 Jan 3 11:21:27 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:27 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:27 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 11:21:27 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:27 service103 kernel: sdq: Write Protect is off Jan 3 11:21:27 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 11:21:27 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 11:21:27 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:27 service103 kernel: sdq: Write Protect is off Jan 3 11:21:27 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 11:21:28 service103 kernel: sdq: unknown partition table Jan 3 11:21:28 service103 run_srp_daemon[24305]: failed srp_daemon: [HCA=mlx4_1] [port=1] [exit status=110]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 11:21:28 service103 kernel: sd 4:0:0:107: Attached scsi disk sdq Jan 3 11:21:28 service103 kernel: sd 4:0:0:107: Attached scsi generic sg21 type 0 Jan 3 11:21:28 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:21:28 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:21:28 service103 kernel: sdr : very big device. try to use READ CAPACITY(16). Jan 3 11:21:28 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:28 service103 kernel: sdr: Write Protect is off Jan 3 11:21:28 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 11:21:28 service103 kernel: sdr : very big device. try to use READ CAPACITY(16). Jan 3 11:21:28 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:21:28 service103 kernel: sdr: Write Protect is off Jan 3 11:21:29 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 11:21:29 service103 kernel: sdr: unknown partition table Jan 3 11:21:29 service103 kernel: sd 4:0:0:115: Attached scsi disk sdr Jan 3 11:21:29 service103 kernel: sd 4:0:0:115: Attached scsi generic sg22 type 0 Jan 3 11:21:34 service103 OpenSM[4756]: SM port is down Jan 3 11:21:37 service103 multipathd: sdad: tur checker reports path is down Jan 3 11:21:38 service103 kernel: host2: ib_srp: Got failed path rec status -110 Jan 3 11:21:38 service103 kernel: host2: ib_srp: Path record query failed Jan 3 11:21:38 service103 kernel: host2: ib_srp: reconnect failed (-110), removing target port. Jan 3 11:21:38 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:38 service103 kernel: end_request: I/O error, dev sds, sector 0 Jan 3 11:21:38 service103 multipathd: sdu: tur checker reports path is down Jan 3 11:21:38 service103 multipathd: sdx: tur checker reports path is down Jan 3 11:21:38 service103 multipathd: sdc: add path (uevent) Jan 3 11:21:39 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:39 service103 kernel: end_request: I/O error, dev sdb, sector 268720136 Jan 3 11:21:39 service103 kernel: device-mapper: multipath: Failing path 8:16. Jan 3 11:21:39 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:39 service103 kernel: end_request: I/O error, dev sdt, sector 13153628272 Jan 3 11:21:40 service103 multipathd: sdb: checker msg is "tur checker reports path is down" Jan 3 11:21:40 service103 kernel: device-mapper: multipath: Failing path 65:48. Jan 3 11:21:41 service103 multipathd: ddn6a-nbp6-ost2: failed to access path sdb Jan 3 11:21:41 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:41 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:32 10] Jan 3 11:21:41 service103 kernel: end_request: I/O error, dev sdy, sector 8053338160 Jan 3 11:21:41 service103 multipathd: 65:48: mark as failed Jan 3 11:21:42 service103 kernel: device-mapper: multipath: Failing path 65:128. Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost26: Entering recovery mode: max_retries=12 Jan 3 11:21:42 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost26: remaining active paths: 0 Jan 3 11:21:42 service103 kernel: end_request: I/O error, dev sdb, sector 268716272 Jan 3 11:21:42 service103 multipathd: 65:32: mark as failed Jan 3 11:21:42 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost18: Entering recovery mode: max_retries=12 Jan 3 11:21:42 service103 kernel: end_request: I/O error, dev sdt, sector 13153624424 Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost18: remaining active paths: 0 Jan 3 11:21:42 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:42 service103 multipathd: 65:96: mark as failed Jan 3 11:21:42 service103 kernel: end_request: I/O error, dev sdy, sector 4496301208 Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost50: Entering recovery mode: max_retries=12 Jan 3 11:21:42 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:42 service103 multipathd: ddn6a-nbp6-ost50: remaining active paths: 0 Jan 3 11:21:42 service103 kernel: end_request: I/O error, dev sdb, sector 268716136 Jan 3 11:21:42 service103 multipathd: 65:160: mark as failed Jan 3 11:21:43 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost82: Entering recovery mode: max_retries=12 Jan 3 11:21:43 service103 kernel: end_request: I/O error, dev sdt, sector 13153622048 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost82: remaining active paths: 0 Jan 3 11:21:43 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:43 service103 multipathd: 65:224: mark as failed Jan 3 11:21:43 service103 kernel: end_request: I/O error, dev sdy, sector 4496299152 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost114: Entering recovery mode: max_retries=12 Jan 3 11:21:43 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost114: remaining active paths: 0 Jan 3 11:21:43 service103 kernel: end_request: I/O error, dev sdb, sector 12288 Jan 3 11:21:43 service103 multipathd: 65:192: mark as failed Jan 3 11:21:43 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost98: Entering recovery mode: max_retries=12 Jan 3 11:21:43 service103 kernel: end_request: I/O error, dev sdt, sector 9059703976 Jan 3 11:21:43 service103 multipathd: ddn6a-nbp6-ost98: remaining active paths: 0 Jan 3 11:21:43 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:43 service103 multipathd: 65:144: mark as failed Jan 3 11:21:44 service103 kernel: end_request: I/O error, dev sdy, sector 4383467472 Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost74: Entering recovery mode: max_retries=12 Jan 3 11:21:44 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost74: remaining active paths: 0 Jan 3 11:21:44 service103 run_srp_daemon[24806]: starting srp_daemon: [HCA=mlx4_1] [port=1] Jan 3 11:21:44 service103 kernel: end_request: I/O error, dev sdb, sector 696 Jan 3 11:21:44 service103 multipathd: 65:80: mark as failed Jan 3 11:21:44 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost42: Entering recovery mode: max_retries=12 Jan 3 11:21:44 service103 kernel: end_request: I/O error, dev sdt, sector 8858390712 Jan 3 11:21:44 service103 OpenSM[4756]: SM port is down Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost42: remaining active paths: 0 Jan 3 11:21:44 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:44 service103 multipathd: 65:176: mark as failed Jan 3 11:21:44 service103 kernel: end_request: I/O error, dev sdy, sector 4362097120 Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost90: Entering recovery mode: max_retries=12 Jan 3 11:21:44 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:44 service103 multipathd: ddn6a-nbp6-ost90: remaining active paths: 0 Jan 3 11:21:44 service103 kernel: end_request: I/O error, dev sdb, sector 0 Jan 3 11:21:45 service103 multipathd: 8:64: mark as failed Jan 3 11:21:45 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:45 service103 multipathd: ddn6a-nbp6-ost10: Entering recovery mode: max_retries=12 Jan 3 11:21:45 service103 kernel: end_request: I/O error, dev sdt, sector 8858390616 Jan 3 11:21:45 service103 multipathd: ddn6a-nbp6-ost10: remaining active paths: 0 Jan 3 11:21:45 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:45 service103 multipathd: sdd: add path (uevent) Jan 3 11:21:45 service103 kernel: end_request: I/O error, dev sdy, sector 4362096984 Jan 3 11:21:45 service103 multipathd: sde: checker msg is "tur checker reports path is down" Jan 3 11:21:45 service103 kernel: scsi 2:0:0:3: rejecting I/O to device being removed Jan 3 11:21:45 service103 multipathd: ddn6a-nbp6-ost10: failed to access path sde Jan 3 11:21:45 service103 kernel: scsi 2:0:0:3: rejecting I/O to device being removed Jan 3 11:21:45 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:48 10] Jan 3 11:21:45 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:21:45 service103 multipathd: 65:128: mark as failed Jan 3 11:21:46 service103 kernel: end_request: I/O error, dev sdb, sector 3098019816 Jan 3 11:21:46 service103 multipathd: ddn6a-nbp6-ost66: Entering recovery mode: max_retries=12 Jan 3 11:21:46 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:46 service103 multipathd: ddn6a-nbp6-ost66: remaining active paths: 0 Jan 3 11:21:46 service103 kernel: end_request: I/O error, dev sdt, sector 8858374568 Jan 3 11:21:46 service103 multipathd: sdf: add path (uevent) Jan 3 11:21:46 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:46 service103 multipathd: sds: checker msg is "tur checker reports path is down" Jan 3 11:21:46 service103 kernel: end_request: I/O error, dev sdy, sector 4362096912 Jan 3 11:21:46 service103 multipathd: ddn6a-nbp6-ost18: failed to access path sds Jan 3 11:21:46 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:46 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:80 10] Jan 3 11:21:46 service103 kernel: end_request: I/O error, dev sdt, sector 8858374552 Jan 3 11:21:46 service103 multipathd: sdg: add path (uevent) Jan 3 11:21:46 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:46 service103 multipathd: sdt: checker msg is "tur checker reports path is down" Jan 3 11:21:46 service103 kernel: end_request: I/O error, dev sdy, sector 4362096840 Jan 3 11:21:46 service103 multipathd: ddn6a-nbp6-ost26: failed to access path sdt Jan 3 11:21:47 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:47 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:96 10] Jan 3 11:21:47 service103 kernel: end_request: I/O error, dev sdt, sector 8858374144 Jan 3 11:21:47 service103 multipathd: sdh: add path (uevent) Jan 3 11:21:47 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:47 service103 multipathd: sdu: checker msg is "tur checker reports path is down" Jan 3 11:21:47 service103 kernel: end_request: I/O error, dev sdy, sector 4362081144 Jan 3 11:21:47 service103 multipathd: ddn6a-nbp6-ost34: failed to access path sdu Jan 3 11:21:47 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:47 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:112 10] Jan 3 11:21:47 service103 kernel: end_request: I/O error, dev sdt, sector 8858372144 Jan 3 11:21:47 service103 multipathd: sdi: add path (uevent) Jan 3 11:21:47 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:47 service103 multipathd: sdv: checker msg is "tur checker reports path is down" Jan 3 11:21:47 service103 kernel: end_request: I/O error, dev sdy, sector 4362081128 Jan 3 11:21:47 service103 multipathd: ddn6a-nbp6-ost42: failed to access path sdv Jan 3 11:21:47 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:47 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:128 10] Jan 3 11:21:47 service103 kernel: end_request: I/O error, dev sdt, sector 2120 Jan 3 11:21:48 service103 multipathd: sdj: add path (uevent) Jan 3 11:21:48 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:48 service103 multipathd: sdw: checker msg is "tur checker reports path is down" Jan 3 11:21:48 service103 kernel: end_request: I/O error, dev sdy, sector 4362081088 Jan 3 11:21:48 service103 multipathd: ddn6a-nbp6-ost50: failed to access path sdw Jan 3 11:21:48 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:48 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:144 10] Jan 3 11:21:48 service103 kernel: end_request: I/O error, dev sdt, sector 0 Jan 3 11:21:48 service103 multipathd: sdk: add path (uevent) Jan 3 11:21:48 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:48 service103 multipathd: sdx: checker msg is "tur checker reports path is down" Jan 3 11:21:48 service103 kernel: end_request: I/O error, dev sdy, sector 4362080256 Jan 3 11:21:48 service103 multipathd: ddn6a-nbp6-ost58: failed to access path sdx Jan 3 11:21:48 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:48 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:160 10] Jan 3 11:21:48 service103 kernel: end_request: I/O error, dev sdt, sector 7516645344 Jan 3 11:21:48 service103 multipathd: sdl: add path (uevent) Jan 3 11:21:48 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:48 service103 multipathd: sdy: checker msg is "tur checker reports path is down" Jan 3 11:21:49 service103 kernel: end_request: I/O error, dev sdy, sector 4362078312 Jan 3 11:21:49 service103 multipathd: ddn6a-nbp6-ost66: failed to access path sdy Jan 3 11:21:49 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:49 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:176 10] Jan 3 11:21:49 service103 kernel: end_request: I/O error, dev sdt, sector 9175039064 Jan 3 11:21:49 service103 multipathd: sdm: add path (uevent) Jan 3 11:21:49 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:49 service103 multipathd: sdz: checker msg is "tur checker reports path is down" Jan 3 11:21:49 service103 kernel: end_request: I/O error, dev sdy, sector 12288 Jan 3 11:21:49 service103 multipathd: ddn6a-nbp6-ost74: failed to access path sdz Jan 3 11:21:49 service103 kernel: sd 2:0:0:27: SCSI error: return code = 0x00010000 Jan 3 11:21:49 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:192 10] Jan 3 11:21:49 service103 kernel: end_request: I/O error, dev sdt, sector 9242986128 Jan 3 11:21:49 service103 multipathd: sdn: add path (uevent) Jan 3 11:21:49 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:49 service103 multipathd: sdaa: checker msg is "tur checker reports path is down" Jan 3 11:21:49 service103 kernel: end_request: I/O error, dev sdy, sector 1048 Jan 3 11:21:49 service103 multipathd: ddn6a-nbp6-ost82: failed to access path sdaa Jan 3 11:21:49 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:50 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:208 10] Jan 3 11:21:50 service103 kernel: end_request: I/O error, dev sdy, sector 0 Jan 3 11:21:50 service103 multipathd: sdo: add path (uevent) Jan 3 11:21:50 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:50 service103 multipathd: sdab: checker msg is "tur checker reports path is down" Jan 3 11:21:50 service103 kernel: end_request: I/O error, dev sdy, sector 8053338368 Jan 3 11:21:50 service103 multipathd: ddn6a-nbp6-ost90: failed to access path sdab Jan 3 11:21:50 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:50 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:224 10] Jan 3 11:21:50 service103 kernel: end_request: I/O error, dev sdy, sector 4499808896 Jan 3 11:21:50 service103 multipathd: sdp: add path (uevent) Jan 3 11:21:50 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:50 service103 multipathd: sdac: checker msg is "tur checker reports path is down" Jan 3 11:21:50 service103 kernel: end_request: I/O error, dev sdy, sector 4512214344 Jan 3 11:21:50 service103 multipathd: ddn6a-nbp6-ost98: failed to access path sdac Jan 3 11:21:50 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:50 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:240 10] Jan 3 11:21:50 service103 kernel: end_request: I/O error, dev sdy, sector 8053346432 Jan 3 11:21:50 service103 multipathd: sdq: add path (uevent) Jan 3 11:21:51 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:51 service103 multipathd: sdad: checker msg is "tur checker reports path is down" Jan 3 11:21:51 service103 kernel: end_request: I/O error, dev sdy, sector 8053346648 Jan 3 11:21:51 service103 multipathd: ddn6a-nbp6-ost106: failed to access path sdad Jan 3 11:21:51 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:51 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:0 10] Jan 3 11:21:51 service103 kernel: end_request: I/O error, dev sdy, sector 8053348488 Jan 3 11:21:51 service103 multipathd: sdr: add path (uevent) Jan 3 11:21:51 service103 kernel: sd 2:0:0:67: SCSI error: return code = 0x00010000 Jan 3 11:21:51 service103 multipathd: sdae: checker msg is "tur checker reports path is down" Jan 3 11:21:51 service103 kernel: end_request: I/O error, dev sdy, sector 8053354640 Jan 3 11:21:51 service103 multipathd: ddn6a-nbp6-ost114: failed to access path sdae Jan 3 11:21:51 service103 kernel: device-mapper: multipath: Failing path 65:32. Jan 3 11:21:51 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:16 10] Jan 3 11:21:51 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:51 service103 multipathd: sdb: remove path (uevent) Jan 3 11:21:51 service103 kernel: end_request: I/O error, dev sds, sector 1714206704 Jan 3 11:21:51 service103 multipathd: sdb: spurious uevent, path not in pathvec Jan 3 11:21:51 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:52 service103 multipathd: uevent trigger error Jan 3 11:21:52 service103 kernel: end_request: I/O error, dev sds, sector 2216423416 Jan 3 11:21:52 service103 multipathd: dm-5: add map (uevent) Jan 3 11:21:52 service103 kernel: scsi 2:0:0:3: rejecting I/O to device being removed Jan 3 11:21:52 service103 multipathd: dm-5: devmap already registered Jan 3 11:21:52 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:52 service103 multipathd: dm-8: add map (uevent) Jan 3 11:21:52 service103 kernel: end_request: I/O error, dev sds, sector 216 Jan 3 11:21:52 service103 multipathd: dm-8: devmap already registered Jan 3 11:21:52 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:52 service103 multipathd: dm-13: add map (uevent) Jan 3 11:21:52 service103 kernel: end_request: I/O error, dev sds, sector 12288 Jan 3 11:21:52 service103 multipathd: dm-13: devmap already registered Jan 3 11:21:52 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:52 service103 multipathd: sde: remove path (uevent) Jan 3 11:21:52 service103 kernel: end_request: I/O error, dev sds, sector 872417384 Jan 3 11:21:52 service103 multipathd: sde: spurious uevent, path not in pathvec Jan 3 11:21:52 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:52 service103 multipathd: uevent trigger error Jan 3 11:21:53 service103 kernel: end_request: I/O error, dev sds, sector 872419328 Jan 3 11:21:53 service103 multipathd: sds: remove path (uevent) Jan 3 11:21:53 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:53 service103 multipathd: sds: spurious uevent, path not in pathvec Jan 3 11:21:53 service103 kernel: end_request: I/O error, dev sds, sector 872420160 Jan 3 11:21:53 service103 multipathd: uevent trigger error Jan 3 11:21:53 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:53 service103 multipathd: sdt: remove path (uevent) Jan 3 11:21:53 service103 kernel: end_request: I/O error, dev sds, sector 872420200 Jan 3 11:21:53 service103 multipathd: sdt: spurious uevent, path not in pathvec Jan 3 11:21:53 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:53 service103 multipathd: uevent trigger error Jan 3 11:21:53 service103 kernel: end_request: I/O error, dev sds, sector 872435920 Jan 3 11:21:53 service103 multipathd: sdu: remove path (uevent) Jan 3 11:21:54 service103 smartd[7498]: Device: /dev/sdb, No such device, open() failed Jan 3 11:21:54 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:54 service103 multipathd: sdu: spurious uevent, path not in pathvec Jan 3 11:21:54 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:21:54 service103 kernel: end_request: I/O error, dev sds, sector 872436168 Jan 3 11:21:54 service103 multipathd: uevent trigger error Jan 3 11:21:54 service103 OpenSM[4756]: SM port is down Jan 3 11:21:54 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:54 service103 multipathd: sdv: remove path (uevent) Jan 3 11:21:54 service103 kernel: end_request: I/O error, dev sds, sector 872436224 Jan 3 11:21:55 service103 multipathd: sdv: spurious uevent, path not in pathvec Jan 3 11:21:55 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:55 service103 multipathd: uevent trigger error Jan 3 11:21:55 service103 kernel: end_request: I/O error, dev sds, sector 939529360 Jan 3 11:21:55 service103 multipathd: sdw: remove path (uevent) Jan 3 11:21:55 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:55 service103 multipathd: uevent trigger error Jan 3 11:21:55 service103 kernel: end_request: I/O error, dev sds, sector 1006638248 Jan 3 11:21:55 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:55 service103 kernel: end_request: I/O error, dev sds, sector 1050375360 Jan 3 11:21:55 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:21:55 service103 kernel: end_request: I/O error, dev sds, sector 4161028264 Jan 3 11:21:55 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:21:55 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:55 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:21:55 service103 kernel: end_request: I/O error, dev sdw, sector 0 Jan 3 11:21:55 service103 smartd[7498]: Device: /dev/sde, No such device, open() failed Jan 3 11:21:55 service103 kernel: device-mapper: multipath: Failing path 65:96. Jan 3 11:21:56 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 11167757376 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 11217928152 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 2456 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 12288 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 10267658488 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 10267660288 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:56 service103 kernel: end_request: I/O error, dev sdw, sector 10267662280 Jan 3 11:21:56 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:57 service103 kernel: end_request: I/O error, dev sdw, sector 10267677224 Jan 3 11:21:57 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:57 service103 kernel: end_request: I/O error, dev sdw, sector 10401879232 Jan 3 11:21:57 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:57 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:21:57 service103 kernel: end_request: I/O error, dev sdw, sector 10401881240 Jan 3 11:21:57 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:21:57 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:57 service103 smartd[7498]: Device: /dev/sds, No such device, open() failed Jan 3 11:21:57 service103 kernel: end_request: I/O error, dev sdw, sector 10791877136 Jan 3 11:21:57 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:21:57 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:57 service103 kernel: end_request: I/O error, dev sdw, sector 15032670264 Jan 3 11:21:58 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdw, sector 15032670432 Jan 3 11:21:58 service103 kernel: sd 2:0:0:51: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdw, sector 15032672312 Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 6181383400 Jan 3 11:21:58 service103 kernel: device-mapper: multipath: Failing path 65:160. Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 7517172864 Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 1496 Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 6176119864 Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 6174020720 Jan 3 11:21:58 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:58 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:21:58 service103 kernel: end_request: I/O error, dev sdaa, sector 6241124904 Jan 3 11:21:59 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:21:59 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:59 service103 smartd[7498]: Device: /dev/sdt, No such device, open() failed Jan 3 11:21:59 service103 kernel: end_request: I/O error, dev sdaa, sector 6375349400 Jan 3 11:21:59 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:21:59 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:59 service103 kernel: end_request: I/O error, dev sdaa, sector 6393693392 Jan 3 11:21:59 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:59 service103 kernel: end_request: I/O error, dev sdaa, sector 12348307768 Jan 3 11:21:59 service103 kernel: sd 2:0:0:83: SCSI error: return code = 0x00010000 Jan 3 11:21:59 service103 kernel: end_request: I/O error, dev sdaa, sector 12348315912 Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 0 Jan 3 11:22:00 service103 kernel: device-mapper: multipath: Failing path 65:192. Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 13696907952 Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 13704938768 Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 3240 Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 3304 Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 12288 Jan 3 11:22:00 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:00 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:00 service103 smartd[7498]: Device: /dev/sdu, No such device, open() failed Jan 3 11:22:00 service103 kernel: end_request: I/O error, dev sdac, sector 1418985448 Jan 3 11:22:01 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:01 service103 kernel: end_request: I/O error, dev sdac, sector 13287831600 Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:01 service103 kernel: end_request: I/O error, dev sdac, sector 13287842096 Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:01 service103 kernel: end_request: I/O error, dev sdac, sector 13287843928 Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:01 service103 kernel: end_request: I/O error, dev sdac, sector 13287845992 Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:01 service103 kernel: end_request: I/O error, dev sdac, sector 13555992632 Jan 3 11:22:01 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdac, sector 13555994624 Jan 3 11:22:02 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdac, sector 13555995088 Jan 3 11:22:02 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdac, sector 13556011304 Jan 3 11:22:02 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdac, sector 13757322416 Jan 3 11:22:02 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdb, sector 3019906200 Jan 3 11:22:02 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:02 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:02 service103 smartd[7498]: Device: /dev/sdv, No such device, open() failed Jan 3 11:22:02 service103 kernel: end_request: I/O error, dev sdb, sector 3019904136 Jan 3 11:22:02 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:02 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sdb, sector 2885702016 Jan 3 11:22:03 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sdb, sector 2885686176 Jan 3 11:22:03 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sdb, sector 2885685248 Jan 3 11:22:03 service103 kernel: scsi 2:0:0:3: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sdb, sector 2885683312 Jan 3 11:22:03 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sds, sector 4161032216 Jan 3 11:22:03 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sds, sector 4161036288 Jan 3 11:22:03 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sds, sector 4161038344 Jan 3 11:22:03 service103 kernel: sd 2:0:0:19: SCSI error: return code = 0x00010000 Jan 3 11:22:03 service103 kernel: end_request: I/O error, dev sds, sector 4161042432 Jan 3 11:22:04 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:04 service103 kernel: end_request: I/O error, dev sdac, sector 13757324424 Jan 3 11:22:04 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:04 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:04 service103 kernel: end_request: I/O error, dev sdac, sector 13824428104 Jan 3 11:22:04 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:04 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:04 service103 smartd[7498]: Device: /dev/sdw, No such device, open() failed Jan 3 11:22:04 service103 kernel: end_request: I/O error, dev sdac, sector 13824430080 Jan 3 11:22:04 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:04 service103 OpenSM[4756]: SM port is down Jan 3 11:22:04 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:04 service103 kernel: end_request: I/O error, dev sdac, sector 13824430688 Jan 3 11:22:05 service103 kernel: sd 2:0:0:99: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdac, sector 13824446544 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 7180655776 Jan 3 11:22:05 service103 kernel: device-mapper: multipath: Failing path 65:224. Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 7180653736 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 6990323384 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 6979342464 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 6979326440 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 6979325952 Jan 3 11:22:05 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:05 service103 kernel: end_request: I/O error, dev sdae, sector 6979323960 Jan 3 11:22:05 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:06 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:06 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:06 service103 kernel: end_request: I/O error, dev sdae, sector 6308526168 Jan 3 11:22:06 service103 smartd[7498]: Device: /dev/sdx, No such device, open() failed Jan 3 11:22:06 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:06 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:06 service103 kernel: end_request: I/O error, dev sdae, sector 6308526136 Jan 3 11:22:06 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:06 service103 kernel: end_request: I/O error, dev sdae, sector 6308524200 Jan 3 11:22:06 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:06 service103 kernel: end_request: I/O error, dev sdae, sector 6308513928 Jan 3 11:22:06 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:06 service103 kernel: end_request: I/O error, dev sdae, sector 12288 Jan 3 11:22:07 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdae, sector 1736 Jan 3 11:22:07 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdae, sector 1672 Jan 3 11:22:07 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdae, sector 0 Jan 3 11:22:07 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdae, sector 7247759432 Jan 3 11:22:07 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdz, sector 0 Jan 3 11:22:07 service103 kernel: device-mapper: multipath: Failing path 65:144. Jan 3 11:22:07 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdz, sector 3927675672 Jan 3 11:22:07 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:07 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:07 service103 smartd[7498]: Device: /dev/sdy, No such device, open() failed Jan 3 11:22:07 service103 kernel: end_request: I/O error, dev sdz, sector 904 Jan 3 11:22:08 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:08 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdz, sector 920 Jan 3 11:22:08 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdz, sector 12288 Jan 3 11:22:08 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdz, sector 3758098440 Jan 3 11:22:08 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdae, sector 7215888064 Jan 3 11:22:08 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdae, sector 7247761408 Jan 3 11:22:08 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:08 service103 kernel: end_request: I/O error, dev sdae, sector 7247762008 Jan 3 11:22:09 service103 kernel: sd 2:0:0:115: SCSI error: return code = 0x00010000 Jan 3 11:22:09 service103 kernel: end_request: I/O error, dev sdae, sector 7247778080 Jan 3 11:22:09 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:09 service103 kernel: end_request: I/O error, dev sdz, sector 3758100560 Jan 3 11:22:09 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:09 service103 kernel: end_request: I/O error, dev sdz, sector 3758117144 Jan 3 11:22:09 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:09 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:09 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:09 service103 kernel: end_request: I/O error, dev sdz, sector 3825207312 Jan 3 11:22:09 service103 smartd[7498]: Device: /dev/sdz, No such device, open() failed Jan 3 11:22:09 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:09 service103 smartd[7498]: Sending warning via mail to root ... Jan 3 11:22:09 service103 kernel: end_request: I/O error, dev sdz, sector 3825207384 Jan 3 11:22:09 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3825209344 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3825209496 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3825210072 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3825226096 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3825226112 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3892319344 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 3959428264 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:10 service103 kernel: end_request: I/O error, dev sdz, sector 4046568016 Jan 3 11:22:10 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 smartd[7498]: Warning via mail to root produced unexpected output (98 bytes) to STDOUT/STDERR: send-mail: fatal: file /etc/postfix/main.cf: parameter setgid_group: unknown group name: postdrop Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 13422055528 Jan 3 11:22:11 service103 smartd[7498]: Warning via mail to root: successful Jan 3 11:22:11 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 13422057512 Jan 3 11:22:11 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 13422057680 Jan 3 11:22:11 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 13422061640 Jan 3 11:22:11 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 13422063912 Jan 3 11:22:11 service103 kernel: sd 2:0:0:75: SCSI error: return code = 0x00010000 Jan 3 11:22:11 service103 kernel: end_request: I/O error, dev sdz, sector 3758100480 Jan 3 11:22:11 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 0 Jan 3 11:22:12 service103 kernel: device-mapper: multipath: Failing path 65:80. Jan 3 11:22:12 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 4575887320 Jan 3 11:22:12 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 4366290776 Jan 3 11:22:12 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 1032 Jan 3 11:22:12 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 1096 Jan 3 11:22:12 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:12 service103 kernel: end_request: I/O error, dev sdv, sector 12288 Jan 3 11:22:13 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:13 service103 kernel: end_request: I/O error, dev sdv, sector 4294969400 Jan 3 11:22:13 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:13 service103 kernel: end_request: I/O error, dev sdv, sector 4294971392 Jan 3 11:22:13 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:13 service103 kernel: end_request: I/O error, dev sdv, sector 4294971856 Jan 3 11:22:13 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:13 service103 kernel: end_request: I/O error, dev sdv, sector 4294971888 Jan 3 11:22:13 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:13 service103 kernel: end_request: I/O error, dev sdv, sector 4294987920 Jan 3 11:22:14 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdv, sector 4294988032 Jan 3 11:22:14 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdv, sector 4297445208 Jan 3 11:22:14 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdv, sector 4496299184 Jan 3 11:22:14 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdv, sector 4496301248 Jan 3 11:22:14 service103 OpenSM[4756]: SM port is down Jan 3 11:22:14 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdv, sector 4563404872 Jan 3 11:22:14 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:14 service103 kernel: end_request: I/O error, dev sdab, sector 7516484296 Jan 3 11:22:14 service103 kernel: device-mapper: multipath: Failing path 65:176. Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 1483552136 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 0 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 1683761368 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 376 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 1476397112 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 1476399104 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:15 service103 kernel: end_request: I/O error, dev sdab, sector 1476399552 Jan 3 11:22:15 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1476415624 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1543506008 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1543507968 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1543508704 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1543524872 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 1677726896 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:16 service103 kernel: end_request: I/O error, dev sdab, sector 9261312088 Jan 3 11:22:16 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sdab, sector 9261316328 Jan 3 11:22:17 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sde, sector 0 Jan 3 11:22:17 service103 kernel: device-mapper: multipath: Failing path 8:64. Jan 3 11:22:17 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sde, sector 7787044416 Jan 3 11:22:17 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sdv, sector 4563406848 Jan 3 11:22:17 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sdv, sector 4563407424 Jan 3 11:22:17 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sdv, sector 4563423336 Jan 3 11:22:17 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:17 service103 kernel: end_request: I/O error, dev sdv, sector 13287839944 Jan 3 11:22:18 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sdv, sector 13287841928 Jan 3 11:22:18 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sdv, sector 13287843856 Jan 3 11:22:18 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sdv, sector 13287844112 Jan 3 11:22:18 service103 kernel: sd 2:0:0:43: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sdv, sector 13287845896 Jan 3 11:22:18 service103 kernel: sd 2:0:0:91: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sdab, sector 9261320280 Jan 3 11:22:18 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sde, sector 1880 Jan 3 11:22:18 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:18 service103 kernel: end_request: I/O error, dev sde, sector 12288 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7784630280 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7784632320 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7784632416 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7784648832 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7851739232 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7851741184 Jan 3 11:22:19 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:19 service103 kernel: end_request: I/O error, dev sde, sector 7851742008 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:20 service103 kernel: end_request: I/O error, dev sde, sector 7851757864 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:20 service103 kernel: end_request: I/O error, dev sde, sector 7985962144 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:20 service103 kernel: end_request: I/O error, dev sde, sector 8124971000 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:20 service103 kernel: end_request: I/O error, dev sde, sector 10335041720 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:20 service103 kernel: end_request: I/O error, dev sde, sector 10335049768 Jan 3 11:22:20 service103 kernel: sd 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:21 service103 kernel: end_request: I/O error, dev sde, sector 10335051992 Jan 3 11:22:21 service103 kernel: scsi 2:0:0:11: SCSI error: return code = 0x00010000 Jan 3 11:22:21 service103 kernel: end_request: I/O error, dev sde, sector 1864 Jan 3 11:22:21 service103 kernel: scsi 2:0:0:11: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: scsi 2:0:0:19: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: Lustre: nbp6-OST0002: slow start_page_write 70s due to heavy IO load Jan 3 11:22:21 service103 kernel: Lustre: nbp6-OST0002: slow start_page_write 70s due to heavy IO load Jan 3 11:22:21 service103 kernel: scsi 2:0:0:27: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: scsi 2:0:0:35: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: scsi 2:0:0:43: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: scsi 2:0:0:51: rejecting I/O to dead device Jan 3 11:22:21 service103 kernel: Lustre: nbp6-OST001a: slow direct_io 75s due to heavy IO load Jan 3 11:22:21 service103 kernel: Lustre: nbp6-OST0022: slow direct_io 80s due to heavy IO load Jan 3 11:22:21 service103 kernel: scsi 2:0:0:59: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:67: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:75: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:83: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:91: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:99: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:107: rejecting I/O to dead device Jan 3 11:22:22 service103 kernel: scsi 2:0:0:115: rejecting I/O to dead device Jan 3 11:22:24 service103 OpenSM[4756]: Entering MASTER state Jan 3 11:22:24 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_1 Jan 3 11:22:24 service103 OpenSM[4756]: SUBNET UP Jan 3 11:22:24 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_1 Jan 3 11:23:36 service103 kernel: scsi5 : SRP.T10:56980F0003C90200 Jan 3 11:23:36 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:36 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 11:23:36 service103 kernel: scsi 5:0:0:0: Attached scsi generic sg4 type 12 Jan 3 11:23:36 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:36 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:36 service103 kernel: sd 5:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:23:36 service103 kernel: sdb: Unit Not Ready, sense: Jan 3 11:23:36 service103 kernel: : Current: sense key: Unit Attention Jan 3 11:23:36 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 11:23:36 service103 kernel: Jan 3 11:23:36 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 11:23:36 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:36 service103 kernel: sdb: Write Protect is off Jan 3 11:23:36 service103 kernel: SCSI device sdb: drive cache: write through w/ FUA Jan 3 11:23:36 service103 multipathd: sdb: add path (uevent) Jan 3 11:23:36 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 11:23:36 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:36 service103 kernel: sdb: Write Protect is off Jan 3 11:23:36 service103 kernel: SCSI device sdb: drive cache: write through w/ FUA Jan 3 11:23:37 service103 kernel: sdb: unknown partition table Jan 3 11:23:37 service103 kernel: sd 5:0:0:3: Attached scsi disk sdb Jan 3 11:23:37 service103 kernel: sd 5:0:0:3: Attached scsi generic sg7 type 0 Jan 3 11:23:37 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:32 10 round-robin 0 1 1 8:16 10] Jan 3 11:23:37 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:37 service103 logger: Adjusted blockdev Jan 3 11:23:37 service103 logger: Adjusted blockdev Jan 3 11:23:37 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:37 service103 multipathd: sde: add path (uevent) Jan 3 11:23:37 service103 logger: Adjusted sdb max_sectors_kb=4096 Jan 3 11:23:37 service103 logger: Adjusted sde max_sectors_kb=4096 Jan 3 11:23:37 service103 logger: Adjusted blockdev Jan 3 11:23:37 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 11:23:37 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:48 10 round-robin 0 1 1 8:64 10] Jan 3 11:23:37 service103 logger: Adjusted sdb scheduler=deadline Jan 3 11:23:37 service103 logger: Adjusted sde scheduler=deadline Jan 3 11:23:37 service103 logger: Adjusted blockdev Jan 3 11:23:37 service103 logger: Adjusted sds max_sectors_kb=4096 Jan 3 11:23:37 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:37 service103 multipathd: sds: add path (uevent) Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjected sdb timeout=280 Jan 3 11:23:38 service103 logger: Adjected sde timeout=280 Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjusted sdt max_sectors_kb=4096 Jan 3 11:23:38 service103 kernel: sde: Write Protect is off Jan 3 11:23:38 service103 logger: Adjusted sds scheduler=deadline Jan 3 11:23:38 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:80 10 round-robin 0 1 1 65:32 10] Jan 3 11:23:38 service103 logger: Adjusted sdu max_sectors_kb=4096 Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjusted sdv max_sectors_kb=4096 Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjusted sdt scheduler=deadline Jan 3 11:23:38 service103 logger: Adjected sds timeout=280 Jan 3 11:23:38 service103 multipathd: sdt: add path (uevent) Jan 3 11:23:38 service103 logger: Adjusted sdu scheduler=deadline Jan 3 11:23:38 service103 logger: Adjusted sdx max_sectors_kb=4096 Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjusted sdv scheduler=deadline Jan 3 11:23:38 service103 logger: Adjusted sdy max_sectors_kb=4096 Jan 3 11:23:38 service103 logger: Adjusted blockdev Jan 3 11:23:38 service103 logger: Adjected sdt timeout=280 Jan 3 11:23:38 service103 kernel: SCSI device sde: drive cache: write through w/ FUA Jan 3 11:23:38 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:96 10 round-robin 0 1 1 65:48 10] Jan 3 11:23:38 service103 logger: Adjected sdu timeout=280 Jan 3 11:23:39 service103 logger: Adjusted sdx scheduler=deadline Jan 3 11:23:39 service103 logger: Adjusted sdz max_sectors_kb=4096 Jan 3 11:23:39 service103 logger: Adjusted blockdev Jan 3 11:23:39 service103 logger: Adjected sdv timeout=280 Jan 3 11:23:39 service103 logger: Adjusted blockdev Jan 3 11:23:39 service103 logger: Adjusted sdy scheduler=deadline Jan 3 11:23:39 service103 logger: Adjusted sdaa max_sectors_kb=4096 Jan 3 11:23:39 service103 logger: Adjusted blockdev Jan 3 11:23:39 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 11:23:39 service103 multipathd: sdu: add path (uevent) Jan 3 11:23:39 service103 logger: Adjected sdx timeout=280 Jan 3 11:23:39 service103 logger: Adjusted blockdev Jan 3 11:23:39 service103 logger: Adjusted sdz scheduler=deadline Jan 3 11:23:39 service103 logger: Adjusted sdac max_sectors_kb=4096 Jan 3 11:23:39 service103 logger: Adjusted sdab max_sectors_kb=4096 Jan 3 11:23:39 service103 logger: Adjected sdy timeout=280 Jan 3 11:23:39 service103 logger: Adjusted sdaa scheduler=deadline Jan 3 11:23:39 service103 logger: Adjusted sdad max_sectors_kb=4096 Jan 3 11:23:39 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:39 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:112 10 round-robin 0 1 1 65:64 10] Jan 3 11:23:39 service103 logger: Adjusted sdae max_sectors_kb=4096 Jan 3 11:23:39 service103 logger: Adjected sdz timeout=280 Jan 3 11:23:39 service103 logger: Adjusted sdac scheduler=deadline Jan 3 11:23:39 service103 logger: Adjusted sdab scheduler=deadline Jan 3 11:23:39 service103 logger: Adjected sdaa timeout=280 Jan 3 11:23:40 service103 logger: Adjusted sdad scheduler=deadline Jan 3 11:23:40 service103 kernel: sde: Write Protect is off Jan 3 11:23:40 service103 multipathd: dm-5: add map (uevent) Jan 3 11:23:40 service103 logger: Adjusted sdae scheduler=deadline Jan 3 11:23:40 service103 logger: Adjected sdac timeout=280 Jan 3 11:23:40 service103 logger: Adjected sdab timeout=280 Jan 3 11:23:40 service103 logger: Adjected sdad timeout=280 Jan 3 11:23:40 service103 multipathd: dm-5: devmap already registered Jan 3 11:23:40 service103 logger: Adjected sdae timeout=280 Jan 3 11:23:40 service103 kernel: SCSI device sde: drive cache: write through w/ FUA Jan 3 11:23:40 service103 multipathd: sdv: add path (uevent) Jan 3 11:23:40 service103 kernel: sde: unknown partition table Jan 3 11:23:40 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:128 10 round-robin 0 1 1 65:80 10] Jan 3 11:23:40 service103 kernel: sd 5:0:0:11: Attached scsi disk sde Jan 3 11:23:40 service103 multipathd: dm-6: add map (uevent) Jan 3 11:23:40 service103 kernel: sd 5:0:0:11: Attached scsi generic sg9 type 0 Jan 3 11:23:40 service103 multipathd: dm-6: devmap already registered Jan 3 11:23:40 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:40 service103 multipathd: dm-7: add map (uevent) Jan 3 11:23:41 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:41 service103 multipathd: dm-7: devmap already registered Jan 3 11:23:41 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 11:23:41 service103 multipathd: sdw: add path (uevent) Jan 3 11:23:41 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:41 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:144 10 round-robin 0 1 1 65:96 10] Jan 3 11:23:41 service103 kernel: sds: Write Protect is off Jan 3 11:23:41 service103 multipathd: dm-8: add map (uevent) Jan 3 11:23:41 service103 multipathd: dm-8: devmap already registered Jan 3 11:23:41 service103 kernel: SCSI device sds: drive cache: write through w/ FUA Jan 3 11:23:41 service103 multipathd: sdx: add path (uevent) Jan 3 11:23:41 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 11:23:41 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:160 10 round-robin 0 1 1 65:112 10] Jan 3 11:23:41 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:41 service103 multipathd: sdy: add path (uevent) Jan 3 11:23:41 service103 kernel: sds: Write Protect is off Jan 3 11:23:41 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:176 10 round-robin 0 1 1 65:128 10] Jan 3 11:23:41 service103 multipathd: dm-9: add map (uevent) Jan 3 11:23:42 service103 kernel: SCSI device sds: drive cache: write through w/ FUA Jan 3 11:23:42 service103 multipathd: dm-9: devmap already registered Jan 3 11:23:42 service103 kernel: sds: unknown partition table Jan 3 11:23:42 service103 multipathd: sdz: add path (uevent) Jan 3 11:23:42 service103 kernel: sd 5:0:0:19: Attached scsi disk sds Jan 3 11:23:42 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:192 10 round-robin 0 1 1 65:144 10] Jan 3 11:23:42 service103 kernel: sd 5:0:0:19: Attached scsi generic sg23 type 0 Jan 3 11:23:42 service103 multipathd: dm-10: add map (uevent) Jan 3 11:23:42 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:42 service103 multipathd: dm-10: devmap already registered Jan 3 11:23:42 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:42 service103 multipathd: sdaa: add path (uevent) Jan 3 11:23:42 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 11:23:42 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:208 10 round-robin 0 1 1 65:160 10] Jan 3 11:23:42 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:42 service103 multipathd: dm-11: add map (uevent) Jan 3 11:23:42 service103 kernel: sdt: Write Protect is off Jan 3 11:23:42 service103 multipathd: dm-11: devmap already registered Jan 3 11:23:42 service103 multipathd: sdab: add path (uevent) Jan 3 11:23:43 service103 kernel: SCSI device sdt: drive cache: write through w/ FUA Jan 3 11:23:43 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:224 10 round-robin 0 1 1 65:176 10] Jan 3 11:23:43 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 11:23:43 service103 multipathd: dm-12: add map (uevent) Jan 3 11:23:43 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:43 service103 multipathd: dm-12: devmap already registered Jan 3 11:23:43 service103 kernel: sdt: Write Protect is off Jan 3 11:23:43 service103 multipathd: sdac: add path (uevent) Jan 3 11:23:43 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:240 10 round-robin 0 1 1 65:192 10] Jan 3 11:23:43 service103 kernel: SCSI device sdt: drive cache: write through w/ FUA Jan 3 11:23:43 service103 multipathd: sdad: add path (uevent) Jan 3 11:23:43 service103 kernel: sdt: unknown partition table Jan 3 11:23:43 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:0 10 round-robin 0 1 1 65:208 10] Jan 3 11:23:43 service103 kernel: sd 5:0:0:27: Attached scsi disk sdt Jan 3 11:23:43 service103 multipathd: dm-13: add map (uevent) Jan 3 11:23:43 service103 kernel: sd 5:0:0:27: Attached scsi generic sg24 type 0 Jan 3 11:23:43 service103 multipathd: dm-13: devmap already registered Jan 3 11:23:43 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:44 service103 multipathd: sdae: add path (uevent) Jan 3 11:23:44 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:44 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:16 10 round-robin 0 1 1 65:224 10] Jan 3 11:23:44 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 11:23:44 service103 multipathd: dm-14: add map (uevent) Jan 3 11:23:44 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:44 service103 multipathd: dm-14: devmap already registered Jan 3 11:23:44 service103 kernel: sdu: Write Protect is off Jan 3 11:23:44 service103 multipathd: dm-0: add map (uevent) Jan 3 11:23:44 service103 multipathd: dm-0: devmap already registered Jan 3 11:23:44 service103 kernel: SCSI device sdu: drive cache: write through w/ FUA Jan 3 11:23:44 service103 multipathd: dm-1: add map (uevent) Jan 3 11:23:44 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 11:23:44 service103 multipathd: dm-1: devmap already registered Jan 3 11:23:44 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:44 service103 multipathd: dm-2: add map (uevent) Jan 3 11:23:44 service103 kernel: sdu: Write Protect is off Jan 3 11:23:44 service103 multipathd: dm-2: devmap already registered Jan 3 11:23:45 service103 multipathd: dm-3: add map (uevent) Jan 3 11:23:45 service103 kernel: SCSI device sdu: drive cache: write through w/ FUA Jan 3 11:23:45 service103 multipathd: dm-3: devmap already registered Jan 3 11:23:45 service103 kernel: sdu: unknown partition table Jan 3 11:23:45 service103 multipathd: dm-4: add map (uevent) Jan 3 11:23:45 service103 kernel: sd 5:0:0:35: Attached scsi disk sdu Jan 3 11:23:45 service103 multipathd: dm-4: devmap already registered Jan 3 11:23:45 service103 kernel: sd 5:0:0:35: Attached scsi generic sg25 type 0 Jan 3 11:23:45 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:45 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:45 service103 kernel: sdv : very big device. try to use READ CAPACITY(16). Jan 3 11:23:45 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:45 service103 kernel: sdv: Write Protect is off Jan 3 11:23:45 service103 kernel: SCSI device sdv: drive cache: write through w/ FUA Jan 3 11:23:45 service103 kernel: sdv : very big device. try to use READ CAPACITY(16). Jan 3 11:23:45 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:45 service103 kernel: sdv: Write Protect is off Jan 3 11:23:46 service103 kernel: SCSI device sdv: drive cache: write through w/ FUA Jan 3 11:23:46 service103 kernel: sdv: unknown partition table Jan 3 11:23:46 service103 kernel: sd 5:0:0:43: Attached scsi disk sdv Jan 3 11:23:46 service103 kernel: sd 5:0:0:43: Attached scsi generic sg26 type 0 Jan 3 11:23:46 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:46 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:46 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 11:23:46 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:46 service103 kernel: sdw: Write Protect is off Jan 3 11:23:46 service103 kernel: SCSI device sdw: drive cache: write through w/ FUA Jan 3 11:23:46 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 11:23:46 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:46 service103 kernel: sdw: Write Protect is off Jan 3 11:23:46 service103 kernel: SCSI device sdw: drive cache: write through w/ FUA Jan 3 11:23:47 service103 kernel: sdw: unknown partition table Jan 3 11:23:47 service103 kernel: sd 5:0:0:51: Attached scsi disk sdw Jan 3 11:23:47 service103 kernel: sd 5:0:0:51: Attached scsi generic sg27 type 0 Jan 3 11:23:47 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:47 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:47 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 11:23:47 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:47 service103 kernel: sdx: Write Protect is off Jan 3 11:23:47 service103 kernel: SCSI device sdx: drive cache: write through w/ FUA Jan 3 11:23:47 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 11:23:47 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:47 service103 kernel: sdx: Write Protect is off Jan 3 11:23:47 service103 kernel: SCSI device sdx: drive cache: write through w/ FUA Jan 3 11:23:48 service103 kernel: sdx: unknown partition table Jan 3 11:23:48 service103 kernel: sd 5:0:0:59: Attached scsi disk sdx Jan 3 11:23:48 service103 kernel: sd 5:0:0:59: Attached scsi generic sg28 type 0 Jan 3 11:23:48 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:48 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:48 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 11:23:48 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:48 service103 kernel: sdy: Write Protect is off Jan 3 11:23:48 service103 kernel: SCSI device sdy: drive cache: write through w/ FUA Jan 3 11:23:48 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 11:23:48 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:48 service103 kernel: sdy: Write Protect is off Jan 3 11:23:49 service103 kernel: SCSI device sdy: drive cache: write through w/ FUA Jan 3 11:23:49 service103 kernel: sdy: unknown partition table Jan 3 11:23:49 service103 kernel: sd 5:0:0:67: Attached scsi disk sdy Jan 3 11:23:49 service103 kernel: sd 5:0:0:67: Attached scsi generic sg29 type 0 Jan 3 11:23:49 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:49 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:49 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 11:23:49 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:49 service103 kernel: sdz: Write Protect is off Jan 3 11:23:49 service103 kernel: SCSI device sdz: drive cache: write through w/ FUA Jan 3 11:23:49 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 11:23:49 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:49 service103 kernel: sdz: Write Protect is off Jan 3 11:23:50 service103 kernel: SCSI device sdz: drive cache: write through w/ FUA Jan 3 11:23:50 service103 kernel: sdz: unknown partition table Jan 3 11:23:50 service103 kernel: sd 5:0:0:75: Attached scsi disk sdz Jan 3 11:23:50 service103 kernel: sd 5:0:0:75: Attached scsi generic sg30 type 0 Jan 3 11:23:50 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:50 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:50 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 11:23:50 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:50 service103 kernel: sdaa: Write Protect is off Jan 3 11:23:50 service103 kernel: SCSI device sdaa: drive cache: write through w/ FUA Jan 3 11:23:50 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 11:23:50 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:50 service103 kernel: sdaa: Write Protect is off Jan 3 11:23:51 service103 kernel: SCSI device sdaa: drive cache: write through w/ FUA Jan 3 11:23:51 service103 kernel: sdaa: unknown partition table Jan 3 11:23:51 service103 kernel: sd 5:0:0:83: Attached scsi disk sdaa Jan 3 11:23:51 service103 kernel: sd 5:0:0:83: Attached scsi generic sg31 type 0 Jan 3 11:23:51 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:51 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:51 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 11:23:51 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:51 service103 kernel: sdab: Write Protect is off Jan 3 11:23:51 service103 kernel: SCSI device sdab: drive cache: write through w/ FUA Jan 3 11:23:51 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 11:23:52 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:52 service103 kernel: sdab: Write Protect is off Jan 3 11:23:52 service103 kernel: SCSI device sdab: drive cache: write through w/ FUA Jan 3 11:23:52 service103 kernel: sdab: unknown partition table Jan 3 11:23:52 service103 kernel: sd 5:0:0:91: Attached scsi disk sdab Jan 3 11:23:52 service103 kernel: sd 5:0:0:91: Attached scsi generic sg32 type 0 Jan 3 11:23:52 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:52 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:52 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 11:23:52 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:53 service103 kernel: sdac: Write Protect is off Jan 3 11:23:53 service103 kernel: SCSI device sdac: drive cache: write through w/ FUA Jan 3 11:23:53 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 11:23:53 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:53 service103 kernel: sdac: Write Protect is off Jan 3 11:23:53 service103 kernel: SCSI device sdac: drive cache: write through w/ FUA Jan 3 11:23:53 service103 kernel: sdac: unknown partition table Jan 3 11:23:53 service103 kernel: sd 5:0:0:99: Attached scsi disk sdac Jan 3 11:23:53 service103 kernel: sd 5:0:0:99: Attached scsi generic sg33 type 0 Jan 3 11:23:53 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:53 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:53 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 11:23:53 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:53 service103 kernel: sdad: Write Protect is off Jan 3 11:23:54 service103 kernel: SCSI device sdad: drive cache: write through w/ FUA Jan 3 11:23:54 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 11:23:54 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:54 service103 kernel: sdad: Write Protect is off Jan 3 11:23:54 service103 kernel: SCSI device sdad: drive cache: write through w/ FUA Jan 3 11:23:54 service103 kernel: sdad: unknown partition table Jan 3 11:23:54 service103 kernel: sd 5:0:0:107: Attached scsi disk sdad Jan 3 11:23:54 service103 kernel: sd 5:0:0:107: Attached scsi generic sg34 type 0 Jan 3 11:23:54 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 11:23:54 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 11:23:54 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 11:23:54 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:54 service103 kernel: sdae: Write Protect is off Jan 3 11:23:55 service103 kernel: SCSI device sdae: drive cache: write through w/ FUA Jan 3 11:23:55 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 11:23:55 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 11:23:55 service103 kernel: sdae: Write Protect is off Jan 3 11:23:55 service103 kernel: SCSI device sdae: drive cache: write through w/ FUA Jan 3 11:23:55 service103 kernel: sdae: unknown partition table Jan 3 11:23:55 service103 kernel: sd 5:0:0:115: Attached scsi disk sdae Jan 3 11:23:55 service103 kernel: sd 5:0:0:115: Attached scsi generic sg35 type 0 Jan 3 11:25:05 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 08d6bf78-0296-bd24-315a-99eaad1b8354 (at 10.151.59.122@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 11:25:05 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 11:25:05 service103 kernel: Lustre: nbp6-OST0072: haven't heard from client 08d6bf78-0296-bd24-315a-99eaad1b8354 (at 10.151.59.122@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 11:25:08 service103 kernel: sd 5:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:08 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:16 10 round-robin 0 1 1 8:32 10] Jan 3 11:25:08 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:64 10 round-robin 0 1 1 8:48 10] Jan 3 11:25:08 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:32 10 round-robin 0 1 1 8:80 10] Jan 3 11:25:08 service103 multipathd: dm-5: add map (uevent) Jan 3 11:25:08 service103 multipathd: dm-5: devmap already registered Jan 3 11:25:08 service103 multipathd: dm-6: add map (uevent) Jan 3 11:25:08 service103 multipathd: dm-6: devmap already registered Jan 3 11:25:08 service103 multipathd: dm-7: add map (uevent) Jan 3 11:25:08 service103 multipathd: dm-7: devmap already registered Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:48 10 round-robin 0 1 1 8:96 10] Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:64 10 round-robin 0 1 1 8:112 10] Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:80 10 round-robin 0 1 1 8:128 10] Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:96 10 round-robin 0 1 1 8:144 10] Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:112 10 round-robin 0 1 1 8:160 10] Jan 3 11:25:09 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:128 10 round-robin 0 1 1 8:176 10] Jan 3 11:25:09 service103 multipathd: dm-8: add map (uevent) Jan 3 11:25:09 service103 multipathd: dm-8: devmap already registered Jan 3 11:25:09 service103 multipathd: dm-9: add map (uevent) Jan 3 11:25:09 service103 multipathd: dm-9: devmap already registered Jan 3 11:25:09 service103 multipathd: dm-10: add map (uevent) Jan 3 11:25:09 service103 multipathd: dm-10: devmap already registered Jan 3 11:25:09 service103 multipathd: dm-11: add map (uevent) Jan 3 11:25:10 service103 kernel: sd 4:0:0:107: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:10 service103 multipathd: dm-11: devmap already registered Jan 3 11:25:10 service103 multipathd: dm-12: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-12: devmap already registered Jan 3 11:25:10 service103 multipathd: dm-13: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-13: devmap already registered Jan 3 11:25:10 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:144 10 round-robin 0 1 1 8:192 10] Jan 3 11:25:10 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:176 10 round-robin 0 1 1 8:224 10] Jan 3 11:25:10 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:192 10 round-robin 0 1 1 8:240 10] Jan 3 11:25:10 service103 kernel: sd 5:0:0:107: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:10 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:208 10 round-robin 0 1 1 65:0 10] Jan 3 11:25:10 service103 multipathd: dm-14: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-14: devmap already registered Jan 3 11:25:10 service103 multipathd: dm-1: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-1: devmap already registered Jan 3 11:25:10 service103 multipathd: dm-2: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-2: devmap already registered Jan 3 11:25:10 service103 multipathd: dm-3: add map (uevent) Jan 3 11:25:10 service103 multipathd: dm-3: devmap already registered Jan 3 11:25:10 service103 kernel: sd 4:0:0:83: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:11 service103 kernel: sd 5:0:0:83: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:12 service103 kernel: sd 4:0:0:115: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:25:14 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:160 10 round-robin 0 1 1 8:208 10] Jan 3 11:25:14 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:224 10 round-robin 0 1 1 65:16 10] Jan 3 11:25:14 service103 multipathd: dm-0: add map (uevent) Jan 3 11:25:14 service103 multipathd: dm-0: devmap already registered Jan 3 11:25:14 service103 multipathd: dm-4: add map (uevent) Jan 3 11:25:14 service103 multipathd: dm-4: devmap already registered Jan 3 11:25:15 service103 kernel: sd 5:0:0:115: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 11:27:21 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.59.81@o2ib [old ver: 12, new ver: 12] Jan 3 11:29:09 service103 kernel: Lustre: nbp6-OST0032: haven't heard from client 2902e15e-0120-99bc-f90d-1166751b4d7d (at 10.151.59.81@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 11:29:09 service103 kernel: Lustre: Skipped 13 previous similar messages Jan 3 12:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 12:42:38 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 15261796-5f57-87b0-e80b-a37b4da84f48 (at 10.151.30.61@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 12:42:38 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 13:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 13:27:21 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.21.242@o2ib [old ver: 12, new ver: 12] Jan 3 13:27:24 service103 kernel: Lustre: 3232:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.0.70@o2ib [old ver: 12, new ver: 12] Jan 3 13:27:26 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.21.241@o2ib [old ver: 12, new ver: 12] Jan 3 13:27:26 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 5 previous similar messages Jan 3 13:27:31 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.2.130@o2ib [old ver: 12, new ver: 12] Jan 3 13:27:31 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 2 previous similar messages Jan 3 13:27:43 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.0.69@o2ib [old ver: 12, new ver: 12] Jan 3 13:27:43 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 13:29:21 service103 kernel: Lustre: nbp6-OST0042: haven't heard from client 8264e2f3-3fba-41a5-4a09-c909e851b932 (at 10.151.21.242@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 13:29:21 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 13:29:21 service103 kernel: Lustre: nbp6-OST0042: haven't heard from client d838e58f-ba66-f2b9-0431-3383503c0bd9 (at 10.151.0.82@o2ib) in 224 seconds. I think it's dead, and I am evicting it. Jan 3 13:48:33 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 302f0e6f-9616-f873-71a2-66c1c398b59d (at 10.151.30.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 13:48:33 service103 kernel: Lustre: Skipped 238 previous similar messages Jan 3 14:10:08 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 14:33:50 service103 kernel: Lustre: Service thread pid 11662 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:33:50 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 14:33:50 service103 kernel: Pid: 11662, comm: ll_ost_io_129 Jan 3 14:33:50 service103 kernel: Jan 3 14:33:50 service103 kernel: Call Trace: Jan 3 14:33:50 service103 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Jan 3 14:33:50 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 14:33:50 service103 kernel: [] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs] Jan 3 14:33:50 service103 kernel: [] filter_commitrw_write+0x1e04/0x2dc0 [obdfilter] Jan 3 14:33:50 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:33:50 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:33:50 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:33:50 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 14:33:51 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:33:51 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:33:51 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:33:51 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:33:51 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:33:51 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:33:51 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:33:51 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:33:51 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:33:52 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:33:52 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:33:52 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:33:52 service103 kernel: Jan 3 14:33:52 service103 kernel: Lustre: Service thread pid 28164 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:33:53 service103 kernel: Pid: 28164, comm: ll_ost_io_337 Jan 3 14:33:53 service103 kernel: Jan 3 14:33:53 service103 kernel: Call Trace: Jan 3 14:33:53 service103 kernel: [] ldiskfs_mb_free_blocks+0x64f/0x710 [ldiskfs] Jan 3 14:33:53 service103 kernel: [] __down_read+0x7a/0x92 Jan 3 14:33:53 service103 kernel: [] down_read+0x11/0x13 Jan 3 14:33:53 service103 kernel: [] __dquot_free_space+0x3d/0x139 Jan 3 14:33:53 service103 kernel: [] dquot_free_space+0xb/0xd Jan 3 14:33:54 service103 kernel: [] ldiskfs_free_blocks+0xa3/0xc0 [ldiskfs] Jan 3 14:33:54 service103 kernel: [] ldiskfs_ext_truncate+0x50a/0xa80 [ldiskfs] Jan 3 14:33:54 service103 kernel: [] wake_up_bit+0x1e/0x23 Jan 3 14:33:54 service103 kernel: [] ldiskfs_truncate+0xb3/0x5c0 [ldiskfs] Jan 3 14:33:54 service103 kernel: [] __getblk+0x25/0x236 Jan 3 14:33:54 service103 kernel: [] __ldiskfs_handle_dirty_metadata+0xdb/0x110 [ldiskfs] Jan 3 14:33:54 service103 kernel: [] unmap_mapping_range+0x59/0x204 Jan 3 14:33:54 service103 kernel: [] ldiskfs_mark_iloc_dirty+0x4a5/0x540 [ldiskfs] Jan 3 14:33:55 service103 kernel: [] vmtruncate+0xa2/0xc9 Jan 3 14:33:55 service103 kernel: [] inode_setattr+0x22/0x104 Jan 3 14:33:55 service103 kernel: [] ldiskfs_setattr+0x2de/0x3a0 [ldiskfs] Jan 3 14:33:55 service103 kernel: [] fsfilt_ldiskfs_setattr+0x1a7/0x250 [fsfilt_ldiskfs] Jan 3 14:33:55 service103 kernel: [] filter_version_get_check+0x91/0x2a0 [obdfilter] Jan 3 14:33:55 service103 kernel: [] up_write+0x9/0xb Jan 3 14:33:55 service103 kernel: [] filter_destroy+0xd9b/0x1fb0 [obdfilter] Jan 3 14:33:55 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 14:33:55 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 14:33:55 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 14:33:55 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:33:55 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:33:55 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:33:55 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] ldlm_lock_put+0x372/0x3d0 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 14:33:56 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 14:33:56 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:33:56 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:33:56 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:33:56 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:33:56 service103 kernel: Jan 3 14:33:57 service103 kernel: Lustre: Service thread pid 28173 was inactive for 200.45s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:33:57 service103 kernel: Pid: 28173, comm: ll_ost_io_346 Jan 3 14:33:57 service103 kernel: Jan 3 14:33:57 service103 kernel: Call Trace: Jan 3 14:33:57 service103 kernel: [] ldiskfs_mb_free_blocks+0x64f/0x710 [ldiskfs] Jan 3 14:33:57 service103 kernel: [] __down_read+0x7a/0x92 Jan 3 14:33:57 service103 kernel: [] down_read+0x11/0x13 Jan 3 14:33:57 service103 kernel: [] __dquot_free_space+0x3d/0x139 Jan 3 14:33:57 service103 kernel: [] dquot_free_space+0xb/0xd Jan 3 14:33:57 service103 kernel: [] ldiskfs_free_blocks+0xa3/0xc0 [ldiskfs] Jan 3 14:33:57 service103 kernel: [] ldiskfs_ext_truncate+0x50a/0xa80 [ldiskfs] Jan 3 14:33:57 service103 kernel: [] wake_up_bit+0x1e/0x23 Jan 3 14:33:57 service103 kernel: [] ldiskfs_truncate+0xb3/0x5c0 [ldiskfs] Jan 3 14:33:58 service103 kernel: [] __getblk+0x25/0x236 Jan 3 14:33:58 service103 kernel: [] __ldiskfs_handle_dirty_metadata+0xdb/0x110 [ldiskfs] Jan 3 14:33:58 service103 kernel: [] unmap_mapping_range+0x59/0x204 Jan 3 14:33:58 service103 kernel: [] ldiskfs_mark_iloc_dirty+0x4a5/0x540 [ldiskfs] Jan 3 14:33:58 service103 kernel: [] vmtruncate+0xa2/0xc9 Jan 3 14:33:58 service103 kernel: [] inode_setattr+0x22/0x104 Jan 3 14:33:58 service103 kernel: [] ldiskfs_setattr+0x2de/0x3a0 [ldiskfs] Jan 3 14:33:58 service103 kernel: [] fsfilt_ldiskfs_setattr+0x1a7/0x250 [fsfilt_ldiskfs] Jan 3 14:33:58 service103 kernel: [] filter_version_get_check+0x91/0x2a0 [obdfilter] Jan 3 14:33:58 service103 kernel: [] up_write+0x9/0xb Jan 3 14:33:58 service103 kernel: [] filter_destroy+0xd9b/0x1fb0 [obdfilter] Jan 3 14:33:58 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 14:33:58 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 14:33:59 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:33:59 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] ldlm_lock_put+0x372/0x3d0 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 14:33:59 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 14:33:59 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:33:59 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:33:59 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:34:00 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:00 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:34:00 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:00 service103 kernel: Jan 3 14:34:00 service103 kernel: Pid: 9365, comm: ll_ost_io_60 Jan 3 14:34:00 service103 kernel: Jan 3 14:34:00 service103 kernel: Call Trace: Jan 3 14:34:00 service103 kernel: [] quota_chk_acq_common+0x127b/0x1340 [lquota] Jan 3 14:34:00 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 14:34:00 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 14:34:00 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 14:34:00 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 14:34:01 service103 kernel: [] fsfilt_ldiskfs_brw_start+0x35c/0x490 [fsfilt_ldiskfs] Jan 3 14:34:01 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 14:34:01 service103 kernel: [] filter_commitrw_write+0xd9b/0x2dc0 [obdfilter] Jan 3 14:34:01 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:01 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:34:01 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:34:01 service103 kernel: [] ldlm_srv_pool_push_slv+0x4c/0x80 [ptlrpc] Jan 3 14:34:01 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:34:01 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:34:01 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:34:01 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:01 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:34:02 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:34:02 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:02 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:02 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:34:02 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:02 service103 kernel: Jan 3 14:34:03 service103 kernel: Lustre: Service thread pid 28262 was inactive for 201.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:34:03 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 14:34:03 service103 kernel: Pid: 28262, comm: ll_ost_io_432 Jan 3 14:34:03 service103 kernel: Jan 3 14:34:03 service103 kernel: Call Trace: Jan 3 14:34:03 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 14:34:03 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 14:34:03 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 14:34:03 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 14:34:03 service103 kernel: [] ldiskfs_acquire_dquot+0x64/0xb0 [ldiskfs] Jan 3 14:34:04 service103 kernel: [] dqget+0x286/0x2b6 Jan 3 14:34:04 service103 kernel: [] dquot_initialize+0x7b/0xac Jan 3 14:34:04 service103 kernel: [] filter_destroy+0x99d/0x1fb0 [obdfilter] Jan 3 14:34:04 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 14:34:04 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:34:04 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] ldlm_lock_put+0x372/0x3d0 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 14:34:04 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 14:34:04 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 14:34:05 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:34:05 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:34:05 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:05 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:34:05 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:05 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:34:05 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:05 service103 kernel: Jan 3 14:34:05 service103 kernel: Lustre: Service thread pid 9418 was inactive for 201.71s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:34:05 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 14:34:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630030.9399 Jan 3 14:34:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18406 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18366 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.9324 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.9371 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18403 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18418 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18401 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18446 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.4575 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.4538 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.18397 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.28267 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630031.28222 Jan 3 14:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.18399 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.28186 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.9311 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.28167 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.18393 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.4574 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.4572 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.9309 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.8449 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.18384 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.18382 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.9405 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.28227 Jan 3 14:34:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630032.28263 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.4571 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28254 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.9331 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.18417 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28206 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.4539 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28261 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.9418 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28262 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.9365 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28173 Jan 3 14:34:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.28164 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630033.11662 Jan 3 14:34:09 service103 kernel: Lustre: Service thread pid 18375 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:34:09 service103 kernel: Lustre: Skipped 35 previous similar messages Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630035.18375 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630035.28248 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630035.18405 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630036.9383 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630036.28161 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630036.18444 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630036.18352 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630036.9321 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630037.8445 Jan 3 14:34:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630038.9361 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630039.28258 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630040.9339 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630040.28149 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630040.9409 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630041.4555 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.28287 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.9426 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.9307 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.28231 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.28180 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630042.9312 Jan 3 14:34:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.26283 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.28983 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.28199 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.9316 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.9385 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.21921 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.8459 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.18392 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.28239 Jan 3 14:34:11 service103 kernel: Lustre: Service thread pid 9327 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:34:11 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.9384 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.4569 Jan 3 14:34:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.9356 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630043.9327 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630045.4545 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630045.28207 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630045.9330 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630047.28156 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630047.28219 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630048.9392 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630048.18355 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630048.28304 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630048.11663 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630049.4553 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630050.18448 Jan 3 14:34:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630050.18404 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630050.4535 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630051.9410 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630053.9389 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630053.9370 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630053.2324 Jan 3 14:34:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630053.9343 Jan 3 14:34:14 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630054.28286 Jan 3 14:34:15 service103 kernel: INFO: task ll_ost_io_02:9307 blocked for more than 120 seconds. Jan 3 14:34:15 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:15 service103 kernel: ll_ost_io_02 D ffff81000903f1a0 0 9307 1 9308 9306 (L-TLB) Jan 3 14:34:15 service103 kernel: ffff810aa68a7780 0000000000000046 ffff81076216d280 ffffffff88b8adbf Jan 3 14:34:15 service103 kernel: 000000030000a068 000000000000000a ffff810aa6841100 ffff810c3fc5c100 Jan 3 14:34:15 service103 kernel: 00084fcc6c654de1 0000000000003fd6 ffff810aa68412e8 000000070000a068 Jan 3 14:34:15 service103 kernel: Call Trace: Jan 3 14:34:15 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xcf/0x210 Jan 3 14:34:15 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:15 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:15 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:15 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:15 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:15 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:34:15 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:34:15 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:15 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:15 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:15 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:15 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:15 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:16 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:16 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:16 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:16 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:16 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:16 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:16 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:17 service103 kernel: Jan 3 14:34:17 service103 kernel: INFO: task ll_ost_io_04:9309 blocked for more than 120 seconds. Jan 3 14:34:17 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:17 service103 kernel: ll_ost_io_04 D ffff810009036b20 0 9309 1 9310 9308 (L-TLB) Jan 3 14:34:17 service103 kernel: ffff810aa68bb9e0 0000000000000046 0000000000000000 ffff810aa68bbbfc Jan 3 14:34:17 service103 kernel: 0000000000000002 000000000000000a ffff810aa6844040 ffff810c3fc26080 Jan 3 14:34:17 service103 kernel: 00084fc9562a325e 000000000000df33 ffff810aa6844228 00000006888f1719 Jan 3 14:34:17 service103 kernel: Call Trace: Jan 3 14:34:17 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:17 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:17 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:17 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:17 service103 kernel: [] :obdfilter:filter_destroy+0x99d/0x1fb0 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:ldlm_blocking_ast+0x0/0x2a0 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:ldlm_completion_ast+0x0/0x880 Jan 3 14:34:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630056.29783 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:ldlm_srv_pool_push_slv+0x4c/0x80 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:lustre_msg_add_version+0x34/0x110 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 14:34:18 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:ldlm_lock_put+0x372/0x3d0 Jan 3 14:34:18 service103 kernel: [] :ost:ost_destroy+0x660/0x790 Jan 3 14:34:18 service103 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Jan 3 14:34:18 service103 kernel: [] :ost:ost_handle+0x1556/0x55b0 Jan 3 14:34:18 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:19 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:19 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:19 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:19 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:19 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:19 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:19 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:19 service103 kernel: Jan 3 14:34:19 service103 kernel: INFO: task ll_ost_io_06:9311 blocked for more than 120 seconds. Jan 3 14:34:19 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:19 service103 kernel: ll_ost_io_06 D 0000000000000000 0 9311 1 9312 9310 (L-TLB) Jan 3 14:34:20 service103 kernel: ffff810aa68d9780 0000000000000046 ffff8106403e21e8 ffffffff88b8ad9a Jan 3 14:34:20 service103 kernel: 000000030000a068 000000000000000a ffff810aa6846080 ffff810723f76080 Jan 3 14:34:20 service103 kernel: 00084fc93103e8f4 00000000000c2c54 ffff810aa6846268 000000070000a068 Jan 3 14:34:20 service103 kernel: Call Trace: Jan 3 14:34:20 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:20 service103 kernel: [] :lquota:quota_is_set+0xf8/0x230 Jan 3 14:34:20 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:20 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:20 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:20 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:20 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:20 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:20 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:21 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:21 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 14:34:21 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:21 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:21 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:21 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:21 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:22 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:22 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:22 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:22 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:22 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:22 service103 kernel: Jan 3 14:34:22 service103 kernel: INFO: task ll_ost_io_07:9312 blocked for more than 120 seconds. Jan 3 14:34:22 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:22 service103 kernel: ll_ost_io_07 D ffff81000903f1a0 0 9312 1 9313 9311 (L-TLB) Jan 3 14:34:22 service103 kernel: ffff810aa68db780 0000000000000046 ffff81074ae58a98 ffffffff88b8ad9a Jan 3 14:34:22 service103 kernel: 0000000300006616 000000000000000a ffff810aa6849820 ffff810c3fc5c100 Jan 3 14:34:22 service103 kernel: 00084fcc8c76d121 00000000000c6225 ffff810aa6849a08 0000000700006616 Jan 3 14:34:23 service103 kernel: Call Trace: Jan 3 14:34:23 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:23 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:23 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:23 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:23 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:23 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:23 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:23 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:23 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:23 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 14:34:23 service103 kernel: [] :ptlrpc:lustre_msg_set_last_committed+0x45/0x120 Jan 3 14:34:23 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:23 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:24 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:24 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:24 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:24 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:24 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:24 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:24 service103 kernel: Jan 3 14:34:25 service103 kernel: INFO: task ll_ost_io_11:9316 blocked for more than 120 seconds. Jan 3 14:34:25 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:25 service103 kernel: ll_ost_io_11 D ffff81000903f1a0 0 9316 1 9317 9315 (L-TLB) Jan 3 14:34:25 service103 kernel: ffff810aa6911780 0000000000000046 ffff81074ae58a98 ffffffff88b8ad9a Jan 3 14:34:25 service103 kernel: 0000000300006616 000000000000000a ffff810aa684e7a0 ffff810c3fc5c100 Jan 3 14:34:25 service103 kernel: 00084fcc8f45ac1e 00000000000c2a34 ffff810aa684e988 0000000700006616 Jan 3 14:34:25 service103 kernel: Call Trace: Jan 3 14:34:25 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:25 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:25 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:25 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:25 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:25 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:26 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:26 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:26 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:26 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:26 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:26 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:26 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:26 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:26 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:26 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:26 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:26 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:26 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:27 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:27 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:27 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:27 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:27 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:27 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:27 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:27 service103 kernel: Jan 3 14:34:27 service103 kernel: INFO: task ll_ost_io_16:9321 blocked for more than 120 seconds. Jan 3 14:34:27 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:27 service103 kernel: ll_ost_io_16 D ffff81000903f1a0 0 9321 1 9322 9320 (L-TLB) Jan 3 14:34:27 service103 kernel: ffff810aa6945780 0000000000000046 ffff810764082bd8 ffffffff88b8ad9a Jan 3 14:34:27 service103 kernel: 000000030000a068 000000000000000a ffff810aa68520c0 ffff810c3fc5c100 Jan 3 14:34:27 service103 kernel: 00084fcb25fae297 0000000000022f5f ffff810aa68522a8 000000070000a068 Jan 3 14:34:28 service103 kernel: Call Trace: Jan 3 14:34:28 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:28 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:28 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:28 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:28 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:28 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:28 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:28 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:28 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:28 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:28 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 14:34:28 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:29 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:29 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:29 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:29 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:29 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:29 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:30 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:30 service103 kernel: Jan 3 14:34:30 service103 kernel: INFO: task ll_ost_io_19:9324 blocked for more than 120 seconds. Jan 3 14:34:30 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:30 service103 kernel: ll_ost_io_19 D ffff81000903f1a0 0 9324 1 9325 9323 (L-TLB) Jan 3 14:34:30 service103 kernel: ffff810aa6985780 0000000000000046 ffff81071ff02a70 ffffffff88b8adbf Jan 3 14:34:30 service103 kernel: 000000030000a068 000000000000000a ffff810aa69547a0 ffff810c3fc5c100 Jan 3 14:34:30 service103 kernel: 00084fc9bb4804cd 0000000000003427 ffff810aa6954988 000000070000a068 Jan 3 14:34:30 service103 kernel: Call Trace: Jan 3 14:34:30 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xcf/0x210 Jan 3 14:34:30 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:30 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:30 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:31 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:31 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:31 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:34:31 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:34:31 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:31 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:31 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:31 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 14:34:31 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:31 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:31 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:31 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:31 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:32 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:32 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:32 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:32 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630058.4554 Jan 3 14:34:32 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:32 service103 kernel: Jan 3 14:34:32 service103 kernel: INFO: task ll_ost_io_22:9327 blocked for more than 120 seconds. Jan 3 14:34:33 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:33 service103 kernel: ll_ost_io_22 D ffff81000903f1a0 0 9327 1 9328 9326 (L-TLB) Jan 3 14:34:33 service103 kernel: ffff810aa69a5780 0000000000000046 ffff810768e18168 ffffffff88b8ad9a Jan 3 14:34:33 service103 kernel: 000000030000a068 000000000000000a ffff810aa6957080 ffff810c3fc5c100 Jan 3 14:34:33 service103 kernel: 00084fccb47544b9 000000000000a1d5 ffff810aa6957268 000000070000a068 Jan 3 14:34:33 service103 kernel: Call Trace: Jan 3 14:34:33 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:33 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:33 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:33 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:33 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:33 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:33 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:34 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:34 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:34 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:34 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:34 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:34 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:34 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:34 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:35 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:35 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:35 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:35 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:35 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:35 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:35 service103 kernel: Jan 3 14:34:35 service103 kernel: INFO: task ll_ost_io_25:9330 blocked for more than 120 seconds. Jan 3 14:34:35 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:35 service103 kernel: ll_ost_io_25 D ffff81000903f1a0 0 9330 1 9331 9329 (L-TLB) Jan 3 14:34:35 service103 kernel: ffff810aa69c5780 0000000000000046 ffff81077bfd7328 ffffffff88b8ad9a Jan 3 14:34:35 service103 kernel: 000000030000a068 000000000000000a ffff810aa695e860 ffff810c3fc5c100 Jan 3 14:34:35 service103 kernel: 00084fcd369ea1c1 00000000000c421c ffff810aa695ea48 000000070000a068 Jan 3 14:34:36 service103 kernel: Call Trace: Jan 3 14:34:36 service103 kernel: [] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210 Jan 3 14:34:36 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:34:36 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:34:36 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:34:36 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:34:36 service103 kernel: [] :obdfilter:filter_commitrw_write+0x93c/0x2dc0 Jan 3 14:34:36 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:34:36 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:36 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:36 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:36 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:36 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:37 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:37 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:37 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:37 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:37 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:37 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:38 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:38 service103 kernel: Jan 3 14:34:38 service103 kernel: INFO: task ll_ost_io_26:9331 blocked for more than 120 seconds. Jan 3 14:34:38 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 14:34:38 service103 kernel: ll_ost_io_26 D ffff81000903f1a0 0 9331 1 9332 9330 (L-TLB) Jan 3 14:34:38 service103 kernel: ffff810aa69cb770 0000000000000046 ffff8108bd9a67a0 5a5a5a5a5a5a5a5a Jan 3 14:34:38 service103 kernel: 5a5a5a5a5a5a5a5a 000000000000000a ffff810aa695e100 ffff810c3fc5c100 Jan 3 14:34:38 service103 kernel: 00084fc8f7a7b9c1 0000000000001039 ffff810aa695e2e8 00000007ffffffff Jan 3 14:34:38 service103 kernel: Call Trace: Jan 3 14:34:38 service103 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Jan 3 14:34:38 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 14:34:38 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0 Jan 3 14:34:38 service103 kernel: [] :obdfilter:filter_commitrw_write+0x1e04/0x2dc0 Jan 3 14:34:38 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:34:39 service103 kernel: [] :obdfilter:filter_commitrw+0x65/0x2c0 Jan 3 14:34:39 service103 kernel: [] :ost:ost_brw_write+0x1c99/0x2480 Jan 3 14:34:39 service103 kernel: [] :obdclass:lh_read_lock+0x13/0x20 Jan 3 14:34:39 service103 kernel: [] :libcfs:cfs_mem_cache_free+0x9/0x10 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 14:34:39 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:34:39 service103 kernel: [] :ost:ost_handle+0x2bae/0x55b0 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:lock_handle_addref+0x5/0x10 Jan 3 14:34:39 service103 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:lock_res_and_lock+0xba/0xd0 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360 Jan 3 14:34:39 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 14:34:40 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 14:34:40 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:34:40 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 14:34:40 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:34:40 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 14:34:40 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:34:40 service103 kernel: Jan 3 14:34:40 service103 kernel: Lustre: Service thread pid 4547 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:34:40 service103 kernel: Lustre: Skipped 24 previous similar messages Jan 3 14:34:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630061.4547 Jan 3 14:34:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630062.28280 Jan 3 14:34:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630062.9390 Jan 3 14:34:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630062.4578 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630064.18358 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630066.18438 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630067.4537 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630070.28212 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630073.18371 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630074.18378 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630075.28259 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630077.28198 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630077.18351 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630080.28292 Jan 3 14:34:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630080.9334 Jan 3 14:34:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630084.18372 Jan 3 14:34:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630085.18354 Jan 3 14:34:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630085.9412 Jan 3 14:34:55 service103 kernel: Lustre: Service thread pid 8451 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:34:55 service103 kernel: Lustre: Skipped 17 previous similar messages Jan 3 14:34:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630095.8451 Jan 3 14:34:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630097.28158 Jan 3 14:34:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630098.18445 Jan 3 14:35:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630108.9366 Jan 3 14:35:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630134.28266 Jan 3 14:35:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630135.17408 Jan 3 14:35:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630135.4577 Jan 3 14:35:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630148.9314 Jan 3 14:35:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630157.18374 Jan 3 14:36:02 service103 kernel: Lustre: Service thread pid 9431 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:36:02 service103 kernel: Lustre: Skipped 8 previous similar messages Jan 3 14:36:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630162.9431 Jan 3 14:36:38 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630198.18423 Jan 3 14:36:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630200.4544 Jan 3 14:40:58 service103 kernel: Lustre: Service thread pid 28230 was inactive for 434.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:40:58 service103 kernel: Pid: 28230, comm: ll_ost_io_400 Jan 3 14:40:58 service103 kernel: Jan 3 14:40:58 service103 kernel: Call Trace: Jan 3 14:40:58 service103 kernel: [] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Jan 3 14:40:58 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:40:58 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:40:58 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:40:58 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:40:58 service103 kernel: [] filter_setattr_internal+0x307/0x1de0 [obdfilter] Jan 3 14:40:58 service103 kernel: [] lookup_one_len+0x53/0x61 Jan 3 14:40:58 service103 kernel: [] filter_parent_unlock+0x14/0x20 [obdfilter] Jan 3 14:40:58 service103 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Jan 3 14:40:59 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 14:40:59 service103 kernel: [] filter_setattr+0x1c1/0x3b0 [obdfilter] Jan 3 14:40:59 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:40:59 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:40:59 service103 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Jan 3 14:40:59 service103 kernel: [] lustre_msg_buf+0x2c/0x90 [ptlrpc] Jan 3 14:40:59 service103 kernel: [] lprocfs_counter_add+0x33/0x100 [lvfs] Jan 3 14:41:00 service103 kernel: [] ost_punch+0x9c8/0xce0 [ost] Jan 3 14:41:00 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] lustre_msg_check_version+0x1e/0x80 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] ost_handle+0x3124/0x55b0 [ost] Jan 3 14:41:00 service103 kernel: [] ldlm_resource_get+0x1d9/0xa60 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:41:00 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:41:00 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:41:00 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:41:01 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:41:01 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:41:01 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:41:01 service103 kernel: Jan 3 14:41:01 service103 kernel: Pid: 9380, comm: ll_ost_io_75 Jan 3 14:41:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630458.28230 Jan 3 14:41:01 service103 kernel: Jan 3 14:41:01 service103 kernel: Call Trace: Jan 3 14:41:01 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:41:01 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:41:01 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:41:01 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:41:01 service103 kernel: [] filter_setattr_internal+0x307/0x1de0 [obdfilter] Jan 3 14:41:02 service103 kernel: [] lookup_one_len+0x53/0x61 Jan 3 14:41:02 service103 kernel: [] filter_parent_unlock+0x14/0x20 [obdfilter] Jan 3 14:41:02 service103 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Jan 3 14:41:02 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 14:41:02 service103 kernel: [] filter_setattr+0x1c1/0x3b0 [obdfilter] Jan 3 14:41:02 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:41:02 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:41:02 service103 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Jan 3 14:41:02 service103 kernel: [] lustre_msg_buf+0x2c/0x90 [ptlrpc] Jan 3 14:41:02 service103 kernel: [] lprocfs_counter_add+0x33/0x100 [lvfs] Jan 3 14:41:02 service103 kernel: [] ost_punch+0x9c8/0xce0 [ost] Jan 3 14:41:02 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:41:02 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 14:41:02 service103 kernel: [] lustre_msg_check_version+0x1e/0x80 [ptlrpc] Jan 3 14:41:03 service103 kernel: [] ost_handle+0x3124/0x55b0 [ost] Jan 3 14:41:03 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:41:03 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:41:03 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:41:03 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:41:03 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:41:03 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:41:03 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:41:03 service103 kernel: Jan 3 14:41:03 service103 kernel: Pid: 18394, comm: ll_ost_io_180 Jan 3 14:41:03 service103 kernel: Jan 3 14:41:03 service103 kernel: Call Trace: Jan 3 14:41:03 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:41:03 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:41:04 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:41:04 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:41:04 service103 kernel: [] filter_setattr_internal+0x307/0x1de0 [obdfilter] Jan 3 14:41:04 service103 kernel: [] lookup_one_len+0x53/0x61 Jan 3 14:41:04 service103 kernel: [] filter_parent_unlock+0x14/0x20 [obdfilter] Jan 3 14:41:04 service103 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Jan 3 14:41:04 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 14:41:04 service103 kernel: [] filter_setattr+0x1c1/0x3b0 [obdfilter] Jan 3 14:41:04 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:41:04 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:41:04 service103 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Jan 3 14:41:04 service103 kernel: [] lustre_msg_buf+0x2c/0x90 [ptlrpc] Jan 3 14:41:04 service103 kernel: [] lprocfs_counter_add+0x33/0x100 [lvfs] Jan 3 14:41:04 service103 kernel: [] ost_punch+0x9c8/0xce0 [ost] Jan 3 14:41:05 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] lustre_msg_check_version+0x1e/0x80 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] ost_handle+0x3124/0x55b0 [ost] Jan 3 14:41:05 service103 kernel: [] move_tasks+0xe7/0x2ec Jan 3 14:41:05 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:41:05 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:41:05 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:41:05 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:41:05 service103 kernel: Jan 3 14:41:05 service103 kernel: Pid: 28283, comm: ll_ost_io_453 Jan 3 14:41:06 service103 kernel: Jan 3 14:41:06 service103 kernel: Call Trace: Jan 3 14:41:06 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:41:06 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:41:06 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:41:06 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:41:06 service103 kernel: [] filter_setattr_internal+0x307/0x1de0 [obdfilter] Jan 3 14:41:06 service103 kernel: [] lookup_one_len+0x53/0x61 Jan 3 14:41:06 service103 kernel: [] filter_parent_unlock+0x14/0x20 [obdfilter] Jan 3 14:41:06 service103 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Jan 3 14:41:06 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 14:41:06 service103 kernel: [] filter_setattr+0x1c1/0x3b0 [obdfilter] Jan 3 14:41:06 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:41:06 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Jan 3 14:41:07 service103 kernel: [] lustre_msg_buf+0x2c/0x90 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] lprocfs_counter_add+0x33/0x100 [lvfs] Jan 3 14:41:07 service103 kernel: [] ost_punch+0x9c8/0xce0 [ost] Jan 3 14:41:07 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] lustre_msg_check_version+0x1e/0x80 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] ost_handle+0x3124/0x55b0 [ost] Jan 3 14:41:07 service103 kernel: [] ldlm_resource_get+0x1d9/0xa60 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:41:07 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:41:07 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:41:08 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:41:08 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:41:08 service103 kernel: Jan 3 14:41:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630459.28283 Jan 3 14:41:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630459.18394 Jan 3 14:41:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630460.9380 Jan 3 14:41:24 service103 kernel: Lustre: Service thread pid 4591 was inactive for 440.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:41:24 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 14:41:24 service103 kernel: Pid: 4591, comm: ll_ost_io_316 Jan 3 14:41:24 service103 kernel: Jan 3 14:41:24 service103 kernel: Call Trace: Jan 3 14:41:24 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:41:24 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:41:24 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:41:24 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:41:24 service103 kernel: [] filter_destroy+0x99d/0x1fb0 [obdfilter] Jan 3 14:41:24 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 14:41:24 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 14:41:24 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] ldlm_lock_put+0x372/0x3d0 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 14:41:24 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 14:41:24 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:41:24 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:41:25 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:41:25 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:41:25 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:41:25 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:41:25 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:41:25 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:41:25 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:41:25 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:41:25 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:41:25 service103 kernel: Jan 3 14:41:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630484.4591 Jan 3 14:41:27 service103 kernel: Lustre: Service thread pid 9416 was inactive for 440.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:41:27 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 14:41:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630487.9416 Jan 3 14:41:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630507.9429 Jan 3 14:41:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630513.28184 Jan 3 14:41:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630517.9354 Jan 3 14:43:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630600.8450 Jan 3 14:43:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630600.28205 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630600.28251 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.9374 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.28204 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.28216 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.28299 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.4579 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.8438 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.9395 Jan 3 14:43:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630601.9372 Jan 3 14:43:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630631.17404 Jan 3 14:43:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630631.8458 Jan 3 14:43:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630638.28195 Jan 3 14:44:10 service103 kernel: Lustre: 9338:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:10 service103 kernel: req@ffff81037069c000 x1387753696564680/t0 o6->ad6b695a-6fca-bf3b-d50b-26454235af95@NET_0x500000a971a10_UUID:0/0 lens 512/400 e 2 to 0 dl 1325630655 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:10 service103 kernel: Lustre: 9338:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Jan 3 14:44:10 service103 kernel: Lustre: 9338:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:10 service103 kernel: req@ffff8108c3da4000 x1387753696564610/t0 o6->ad6b695a-6fca-bf3b-d50b-26454235af95@NET_0x500000a971a10_UUID:0/0 lens 512/400 e 2 to 0 dl 1325630655 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:11 service103 kernel: Lustre: 9401:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:11 service103 kernel: req@ffff810a8823d400 x1386958894358190/t0 o6->12cb12da-9cd9-4a49-9344-c293ad1f9e71@NET_0x500000a972a02_UUID:0/0 lens 512/400 e 2 to 0 dl 1325630656 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:11 service103 kernel: Lustre: 9401:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Jan 3 14:44:13 service103 kernel: Lustre: 9368:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:13 service103 kernel: req@ffff81064633dc50 x1390004822228432/t0 o4->f8927383-08ce-6e77-a5d1-42c743792851@NET_0x500000a97355e_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630658 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:13 service103 kernel: Lustre: 9368:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Jan 3 14:44:16 service103 kernel: Lustre: 4559:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:16 service103 kernel: req@ffff81096a11d800 x1387620706451141/t0 o4->5e9e2c73-aa6a-04e9-d183-0a3905e13187@NET_0x500000a972a10_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630661 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:16 service103 kernel: Lustre: 4559:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages Jan 3 14:44:21 service103 kernel: Lustre: 28253:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 14:44:21 service103 kernel: req@ffff810bdc5f6400 x1390004814882359/t0 o4->dc4d266f-a615-98e7-905d-ba4b335be73e@NET_0x500000a973585_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630666 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:21 service103 kernel: Lustre: 28253:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages Jan 3 14:44:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325630671.5576 Jan 3 14:44:32 service103 kernel: Lustre: 28202:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-231), not sending early reply Jan 3 14:44:32 service103 kernel: req@ffff810422282000 x1390004814890112/t0 o4->61f2b20d-b871-39b2-363d-284f27370a17@NET_0x500000a973577_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630677 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:32 service103 kernel: Lustre: 28202:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 20 previous similar messages Jan 3 14:44:50 service103 kernel: Lustre: 11668:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-232), not sending early reply Jan 3 14:44:50 service103 kernel: req@ffff81036d87f800 x1387620706451661/t0 o4->5e9e2c73-aa6a-04e9-d183-0a3905e13187@NET_0x500000a972a10_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630695 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:44:50 service103 kernel: Lustre: 11668:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 24 previous similar messages Jan 3 14:45:27 service103 kernel: Lustre: 9364:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-232), not sending early reply Jan 3 14:45:27 service103 kernel: req@ffff8108f39d3c00 x1390004814883423/t0 o4->dc4d266f-a615-98e7-905d-ba4b335be73e@NET_0x500000a973585_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630732 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:45:27 service103 kernel: Lustre: 9364:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 20 previous similar messages Jan 3 14:46:34 service103 kernel: Lustre: 28265:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-232), not sending early reply Jan 3 14:46:34 service103 kernel: req@ffff8104e4c0dc00 x1390004820146683/t0 o4->a30eccc5-95f4-5476-6077-7042d2a84fa8@NET_0x500000a973586_UUID:0/0 lens 448/416 e 2 to 0 dl 1325630799 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:46:34 service103 kernel: Lustre: 28265:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 19 previous similar messages Jan 3 14:47:38 service103 kernel: Lustre: 9196:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:47:38 service103 kernel: Lustre: 9196:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff8109bff08c00; still busy with 1 active RPCs Jan 3 14:47:38 service103 kernel: Lustre: 9196:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages Jan 3 14:47:38 service103 kernel: LustreError: 9196:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b78cd1400 x1389011668531124/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325630958 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:47:39 service103 kernel: LustreError: 9196:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 45 previous similar messages Jan 3 14:47:45 service103 kernel: Lustre: 9212:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:47:45 service103 kernel: LustreError: 9212:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810473878000 x1389011668531246/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325630965 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:47:52 service103 kernel: Lustre: 14532:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:47:52 service103 kernel: LustreError: 14532:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8103e6ebe000 x1389011668531248/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325630972 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:47:59 service103 kernel: Lustre: 9206:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:47:59 service103 kernel: LustreError: 9206:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810ac9e2dc00 x1389011668531250/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325630979 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:48:06 service103 kernel: Lustre: 6601:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:48:13 service103 kernel: Lustre: 9226:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:48:13 service103 kernel: LustreError: 9226:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81086d12d400 x1389011668531373/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325630993 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:48:14 service103 kernel: LustreError: 9226:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jan 3 14:48:27 service103 kernel: Lustre: 30366:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:48:27 service103 kernel: Lustre: 30366:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Jan 3 14:48:34 service103 kernel: LustreError: 9290:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810370ad5800 x1389011668531498/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325631014 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:48:34 service103 kernel: LustreError: 9290:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Jan 3 14:48:45 service103 kernel: Lustre: 8299:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: fa21cf93-a978-6f00-557e-1048c45b07d4 reconnecting Jan 3 14:48:45 service103 kernel: Lustre: 8299:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 26 previous similar messages Jan 3 14:48:53 service103 kernel: Lustre: 26509:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-95), not sending early reply Jan 3 14:48:53 service103 kernel: req@ffff8108e02abc00 x1387390898963185/t0 o101->5be582e6-a6f3-3101-5e80-af0c84cb8e6e@NET_0x500000a9719d1_UUID:0/0 lens 296/0 e 1 to 0 dl 1325630937 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:48:53 service103 kernel: Lustre: 26509:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Jan 3 14:48:54 service103 kernel: Lustre: 16750:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from c3b1fde9-c48c-6ba9-d673-eb4737e5c821@10.151.32.151@o2ib to 0xffff810a089ad200; still busy with 1 active RPCs Jan 3 14:48:54 service103 kernel: Lustre: 16750:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 46 previous similar messages Jan 3 14:49:13 service103 kernel: LustreError: 21328:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810adbf11c00 x1386957194140538/t0 o8->4e6c4b7c-92ef-e769-2335-40ef60662fe4@NET_0x500000a973b4c_UUID:0/0 lens 368/264 e 0 to 0 dl 1325631053 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:49:13 service103 kernel: LustreError: 21328:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 63 previous similar messages Jan 3 14:49:18 service103 kernel: Lustre: 10623:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: 0df8d608-e399-bae5-65d5-b6830a1b1e9e reconnecting Jan 3 14:49:18 service103 kernel: Lustre: 10623:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 42 previous similar messages Jan 3 14:50:25 service103 kernel: Lustre: 14532:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: nbp6-mdtlov_UUID reconnecting Jan 3 14:50:25 service103 kernel: Lustre: 14532:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 26 previous similar messages Jan 3 14:50:32 service103 kernel: LustreError: 9235:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810bcaf6c400 x1389011668532667/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325631132 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:50:32 service103 kernel: LustreError: 9235:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 33 previous similar messages Jan 3 14:51:25 service103 kernel: Lustre: 14530:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from 70575fa9-c61f-acc6-c85e-5335dc36a08b@10.151.53.61@o2ib to 0xffff810a39c20400; still busy with 1 active RPCs Jan 3 14:51:25 service103 kernel: Lustre: 14530:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 112 previous similar messages Jan 3 14:51:35 service103 kernel: Lustre: Service thread pid 8454 was inactive for 858.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:51:35 service103 kernel: Pid: 8454, comm: ll_ost_io_506 Jan 3 14:51:35 service103 kernel: Jan 3 14:51:35 service103 kernel: Call Trace: Jan 3 14:51:35 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:51:35 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:51:35 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:51:35 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:51:35 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:51:35 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 14:51:35 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:51:35 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:51:35 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:51:35 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:51:35 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:51:35 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:51:35 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:51:36 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:51:36 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:51:36 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:51:36 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:51:36 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:51:36 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:51:37 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:51:37 service103 kernel: Jan 3 14:51:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631096.8454 Jan 3 14:51:37 service103 kernel: Pid: 4536, comm: ll_ost_io_266 Jan 3 14:51:37 service103 kernel: Jan 3 14:51:37 service103 kernel: Call Trace: Jan 3 14:51:37 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:51:37 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:51:37 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:51:37 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:51:37 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:51:37 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:51:37 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:51:38 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:51:38 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:51:38 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:51:38 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:51:38 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:51:38 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:51:38 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:51:38 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:51:38 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:51:38 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:51:38 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:51:38 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:51:38 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:51:38 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:51:38 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:51:39 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:51:39 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:51:39 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:51:39 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:51:39 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:51:39 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:51:40 service103 kernel: Jan 3 14:51:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631096.4536 Jan 3 14:52:33 service103 kernel: Lustre: 9210:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 14:52:33 service103 kernel: Lustre: 9210:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 124 previous similar messages Jan 3 14:52:51 service103 kernel: Lustre: Service thread pid 4541 was inactive for 870.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:52:51 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 14:52:51 service103 kernel: Pid: 4541, comm: ll_ost_io_271 Jan 3 14:52:51 service103 kernel: Jan 3 14:52:51 service103 kernel: Call Trace: Jan 3 14:52:51 service103 kernel: [] ldiskfs_get_blocks+0xcf/0x210 [ldiskfs] Jan 3 14:52:51 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:52:51 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:52:51 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:52:51 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:52:51 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 14:52:51 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:52:51 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:52:51 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:52:51 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:52:51 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:52:51 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:52:51 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:52:51 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:52:51 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:52:51 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:52:51 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:52:51 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:52:51 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:52:52 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:52:52 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:52:52 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:52:52 service103 kernel: Jan 3 14:52:52 service103 kernel: Pid: 9375, comm: ll_ost_io_70 Jan 3 14:52:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631171.4541 Jan 3 14:52:52 service103 kernel: Jan 3 14:52:53 service103 kernel: Call Trace: Jan 3 14:52:53 service103 kernel: [] ldiskfs_get_blocks+0xcf/0x210 [ldiskfs] Jan 3 14:52:53 service103 kernel: [] filter_quota_check+0x0/0xb0 [lquota] Jan 3 14:52:53 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:52:53 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:52:53 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:52:53 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:52:53 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:52:53 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 14:52:53 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:52:53 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:52:53 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:52:53 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:52:53 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:52:54 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:52:54 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:52:54 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:52:54 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:52:54 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:52:55 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:52:55 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:52:55 service103 kernel: Jan 3 14:52:55 service103 kernel: Pid: 28168, comm: ll_ost_io_341 Jan 3 14:52:55 service103 kernel: Jan 3 14:52:55 service103 kernel: Call Trace: Jan 3 14:52:55 service103 kernel: [] ldiskfs_get_blocks+0xcf/0x210 [ldiskfs] Jan 3 14:52:55 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:52:55 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 14:52:55 service103 kernel: [] __down_write+0xb/0xd Jan 3 14:52:55 service103 kernel: [] down_write+0x11/0x13 Jan 3 14:52:55 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 14:52:55 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 14:52:56 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:52:56 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:52:56 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:52:56 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:52:56 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:52:56 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:52:56 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:52:56 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:52:56 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:52:57 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:52:57 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:52:57 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:52:57 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:52:57 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:52:57 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:52:57 service103 kernel: Jan 3 14:52:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631172.28168 Jan 3 14:52:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631172.9375 Jan 3 14:52:57 service103 kernel: Lustre: Service thread pid 28244 was inactive for 870.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 14:52:57 service103 kernel: Lustre: Skipped 18 previous similar messages Jan 3 14:52:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631172.28244 Jan 3 14:53:06 service103 kernel: LustreError: 9260:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81049af47000 x1389011668536822/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325631286 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:53:06 service103 kernel: LustreError: 9260:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 135 previous similar messages Jan 3 14:53:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631205.28182 Jan 3 14:53:44 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f004327b-0abe-aae7-69bd-fc14a25b36c6 (at 10.151.41.106@o2ib) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 14:53:44 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 14:53:55 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 0d6ce5d2-efdc-7f07-530d-4937fd2e18e3 (at 10.151.41.113@o2ib) in 156 seconds. I think it's dead, and I am evicting it. Jan 3 14:54:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631263.8455 Jan 3 14:54:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631265.4540 Jan 3 14:54:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631266.4586 Jan 3 14:54:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631281.10624 Jan 3 14:56:20 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f004327b-0abe-aae7-69bd-fc14a25b36c6 (at (no nid)) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 14:56:20 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 14:56:25 service103 kernel: Lustre: 30994:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from 90c38ae2-5d75-7ff7-82bb-0121ce54d282@10.151.42.14@o2ib to 0xffff81098cd5ae00; still busy with 6 active RPCs Jan 3 14:56:25 service103 kernel: Lustre: 30994:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 242 previous similar messages Jan 3 14:56:31 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fa21cf93-a978-6f00-557e-1048c45b07d4 (at (no nid)) in 154 seconds. I think it's dead, and I am evicting it. Jan 3 14:56:50 service103 kernel: Lustre: 6604:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: b9e71ba3-c238-1a67-05f4-07f676317798 reconnecting Jan 3 14:56:50 service103 kernel: Lustre: 6604:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 215 previous similar messages Jan 3 14:57:22 service103 kernel: Lustre: 16752:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jan 3 14:57:22 service103 kernel: req@ffff81063fb66c00 x1387481519647855/t0 o101->55292703-8b92-17f7-7381-401f6284429a@NET_0x500000a9719d3_UUID:0/0 lens 296/0 e 0 to 0 dl 1325631446 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 14:57:22 service103 kernel: Lustre: 16752:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jan 3 14:57:31 service103 kernel: Lustre: Service thread pid 9363 was inactive for 870.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 14:57:31 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 14:57:31 service103 kernel: Pid: 9363, comm: ll_ost_io_58 Jan 3 14:57:31 service103 kernel: Jan 3 14:57:31 service103 kernel: Call Trace: Jan 3 14:57:31 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:57:31 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:57:31 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:57:31 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:57:31 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:57:31 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:57:31 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:57:31 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:57:31 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:57:31 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:57:31 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:57:31 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:31 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:57:31 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:31 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:57:31 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:57:31 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:57:31 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:57:32 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:57:32 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:57:32 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:57:32 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:57:32 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:57:32 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:57:33 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:57:33 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:57:33 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:57:33 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:57:33 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:57:33 service103 kernel: Jan 3 14:57:33 service103 kernel: Pid: 28150, comm: ll_ost_io_323 Jan 3 14:57:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631451.9363 Jan 3 14:57:33 service103 kernel: Jan 3 14:57:33 service103 kernel: Call Trace: Jan 3 14:57:33 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:57:33 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:57:33 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:57:33 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:57:34 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:57:34 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:57:34 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:57:34 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:57:34 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:57:34 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:57:34 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:57:35 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:35 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:57:35 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:35 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:57:35 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:57:35 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:57:35 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:57:35 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:57:35 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:57:35 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:57:35 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:57:35 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:57:35 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:57:35 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:57:36 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:57:36 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:57:36 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:57:36 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:57:36 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:57:36 service103 kernel: Jan 3 14:57:36 service103 kernel: Pid: 9369, comm: ll_ost_io_64 Jan 3 14:57:37 service103 kernel: Jan 3 14:57:37 service103 kernel: Call Trace: Jan 3 14:57:37 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:57:37 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:57:37 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:57:37 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:57:37 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:57:37 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:57:37 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:57:37 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:57:37 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:57:37 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:57:37 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:57:37 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:57:38 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:57:38 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:38 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:57:38 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:38 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:57:38 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:57:38 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:57:38 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:57:38 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:57:38 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:57:38 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:57:38 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:57:38 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:57:38 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:57:39 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:57:39 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:57:39 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:57:39 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:57:39 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:57:39 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:57:39 service103 kernel: Jan 3 14:57:39 service103 kernel: Pid: 9394, comm: ll_ost_io_89 Jan 3 14:57:40 service103 kernel: Jan 3 14:57:40 service103 kernel: Call Trace: Jan 3 14:57:40 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:57:40 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 14:57:40 service103 kernel: [] find_busiest_group+0x20d/0x621 Jan 3 14:57:40 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:57:40 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:57:40 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:57:40 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:57:40 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:57:40 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:57:40 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:57:40 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:57:41 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:57:41 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:57:41 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:41 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:57:41 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:41 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:57:41 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:57:41 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:57:41 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:57:41 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:57:41 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:57:41 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:57:41 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:57:42 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:57:42 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:57:42 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:57:42 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:57:42 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:57:42 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:57:42 service103 kernel: Jan 3 14:57:42 service103 kernel: Pid: 28270, comm: ll_ost_io_440 Jan 3 14:57:43 service103 kernel: Jan 3 14:57:43 service103 kernel: Call Trace: Jan 3 14:57:43 service103 kernel: [] generic_make_request+0x236/0x24d Jan 3 14:57:43 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 14:57:43 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 14:57:43 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 14:57:43 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 14:57:43 service103 kernel: [] ldiskfs_ext_get_blocks+0xd2/0x1840 [ldiskfs] Jan 3 14:57:43 service103 kernel: [] compute_remquota+0x366/0x6f0 [lquota] Jan 3 14:57:43 service103 kernel: [] quota_chk_acq_common+0xbd0/0x1340 [lquota] Jan 3 14:57:43 service103 kernel: [] lustre_hash_lookup+0x228/0x2b0 [obdclass] Jan 3 14:57:43 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 14:57:43 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 14:57:43 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:44 service103 kernel: [] filter_quota_check+0x81/0xb0 [lquota] Jan 3 14:57:44 service103 kernel: [] filter_quota_acquire+0x0/0x120 [lquota] Jan 3 14:57:44 service103 kernel: [] filter_commitrw_write+0x7dd/0x2dc0 [obdfilter] Jan 3 14:57:44 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 14:57:44 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 14:57:44 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 14:57:44 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 14:57:44 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 14:57:44 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 14:57:44 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 14:57:44 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 14:57:44 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 14:57:44 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 14:57:45 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 14:57:45 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 14:57:45 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] child_rip+0xa/0x11 Jan 3 14:57:45 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 14:57:45 service103 kernel: [] child_rip+0x0/0x11 Jan 3 14:57:45 service103 kernel: Jan 3 14:57:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.28213 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.28257 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.9391 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.28270 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.9394 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.9369 Jan 3 14:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631454.28150 Jan 3 14:58:07 service103 kernel: LustreError: 30366:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8103f3bd1800 x1389011668538618/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325631587 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 14:58:07 service103 kernel: LustreError: 30366:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 268 previous similar messages Jan 3 14:58:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631497.9335 Jan 3 14:58:56 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f004327b-0abe-aae7-69bd-fc14a25b36c6 (at (no nid)) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 14:58:56 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 14:59:07 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 0d6ce5d2-efdc-7f07-530d-4937fd2e18e3 (at (no nid)) in 156 seconds. I think it's dead, and I am evicting it. Jan 3 14:59:07 service103 kernel: Lustre: 9256:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810a2274fa00 already connecting Jan 3 15:01:32 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f004327b-0abe-aae7-69bd-fc14a25b36c6 (at (no nid)) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 15:01:32 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 15:03:04 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 5bdd42c7-c965-e756-f108-d718f4a68e6c (at 10.151.33.20@o2ib) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 15:03:04 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 15:04:08 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f004327b-0abe-aae7-69bd-fc14a25b36c6 (at (no nid)) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 15:04:19 service103 kernel: Lustre: 8299:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810a3094aa00 already connecting Jan 3 15:04:52 service103 kernel: Lustre: Service thread pid 24866 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:04:52 service103 kernel: Lustre: Skipped 4 previous similar messages Jan 3 15:04:52 service103 kernel: Pid: 24866, comm: ll_ost_159 Jan 3 15:04:52 service103 kernel: Jan 3 15:04:52 service103 kernel: Call Trace: Jan 3 15:04:52 service103 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Jan 3 15:04:52 service103 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Jan 3 15:04:52 service103 kernel: [] __down+0xc3/0xd8 Jan 3 15:04:52 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:04:52 service103 kernel: [] __down_failed+0x35/0x3a Jan 3 15:04:52 service103 kernel: [] .text.lock.ldlm_resource+0x41/0x87 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:04:52 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ldlm_lock_create+0xba/0x9f0 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] lustre_swab_reqbuf+0xfb/0x120 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:04:52 service103 kernel: [] ldlm_handle_enqueue+0x66f/0x1210 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Jan 3 15:04:52 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:04:52 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:04:52 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:04:53 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:04:53 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:04:53 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:04:53 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:04:53 service103 kernel: Jan 3 15:04:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631892.24866 Jan 3 15:05:23 service103 kernel: Lustre: 9187:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: nbp6-mdtlov_UUID reconnecting Jan 3 15:05:23 service103 kernel: Lustre: 9187:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 462 previous similar messages Jan 3 15:05:40 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 5bdd42c7-c965-e756-f108-d718f4a68e6c (at (no nid)) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 15:05:40 service103 kernel: Lustre: Skipped 4 previous similar messages Jan 3 15:06:06 service103 kernel: Pid: 9353, comm: ll_ost_io_48 Jan 3 15:06:06 service103 kernel: Jan 3 15:06:06 service103 kernel: Call Trace: Jan 3 15:06:07 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 15:06:07 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 15:06:07 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 15:06:07 service103 kernel: [] __down_write+0xb/0xd Jan 3 15:06:07 service103 kernel: [] down_write+0x11/0x13 Jan 3 15:06:07 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 15:06:07 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 15:06:07 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 15:06:07 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:06:07 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 15:06:08 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 15:06:08 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:06:08 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 15:06:08 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 15:06:08 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 15:06:08 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:06:09 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:06:09 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:06:09 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:06:09 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:06:09 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:06:09 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:06:09 service103 kernel: Jan 3 15:06:09 service103 kernel: Pid: 28249, comm: ll_ost_io_419 Jan 3 15:06:09 service103 kernel: Jan 3 15:06:09 service103 kernel: Call Trace: Jan 3 15:06:09 service103 kernel: [] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs] Jan 3 15:06:09 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 15:06:09 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 15:06:10 service103 kernel: [] __down_write+0xb/0xd Jan 3 15:06:10 service103 kernel: [] down_write+0x11/0x13 Jan 3 15:06:10 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 15:06:10 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 15:06:10 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 15:06:10 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:06:10 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 15:06:10 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 15:06:10 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:06:10 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 15:06:10 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 15:06:10 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:06:10 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:06:11 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 15:06:11 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 15:06:11 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:06:11 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:06:11 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:06:11 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:06:11 service103 kernel: Jan 3 15:06:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631967.28249 Jan 3 15:06:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325631968.9353 Jan 3 15:06:26 service103 kernel: Lustre: 9191:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff8109bff08c00; still busy with 1 active RPCs Jan 3 15:06:26 service103 kernel: Lustre: 9191:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 536 previous similar messages Jan 3 15:08:08 service103 kernel: LustreError: 4471:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c0bec6400 x1387753474668390/t0 o8->cdae5e50-9934-521e-617a-71f5627bce49@NET_0x500000a971a19_UUID:0/0 lens 368/264 e 0 to 0 dl 1325632188 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 15:08:08 service103 kernel: LustreError: 4471:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 545 previous similar messages Jan 3 15:08:16 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 5bdd42c7-c965-e756-f108-d718f4a68e6c (at (no nid)) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 15:08:16 service103 kernel: Lustre: Skipped 4 previous similar messages Jan 3 15:08:25 service103 kernel: Lustre: Service thread pid 9294 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:08:25 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 15:08:25 service103 kernel: Pid: 9294, comm: ll_ost_119 Jan 3 15:08:25 service103 kernel: Jan 3 15:08:25 service103 kernel: Call Trace: Jan 3 15:08:25 service103 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Jan 3 15:08:25 service103 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Jan 3 15:08:25 service103 kernel: [] __down+0xc3/0xd8 Jan 3 15:08:25 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:08:25 service103 kernel: [] __down_failed+0x35/0x3a Jan 3 15:08:25 service103 kernel: [] .text.lock.ldlm_resource+0x41/0x87 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:08:25 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ldlm_lock_create+0xba/0x9f0 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] lustre_swab_reqbuf+0xfb/0x120 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:08:25 service103 kernel: [] ldlm_handle_enqueue+0x66f/0x1210 [ptlrpc] Jan 3 15:08:25 service103 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Jan 3 15:08:25 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:08:26 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:08:26 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:08:26 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:08:26 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:08:26 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:08:26 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:08:26 service103 kernel: Jan 3 15:08:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632105.9294 Jan 3 15:09:25 service103 kernel: Lustre: 10627:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0062: exp ffff81097ca6c000 already connecting Jan 3 15:09:29 service103 kernel: Pid: 18413, comm: ll_ost_io_199 Jan 3 15:09:29 service103 kernel: Jan 3 15:09:29 service103 kernel: Call Trace: Jan 3 15:09:29 service103 kernel: [] ldlm_lock_remove_from_lru+0x74/0xe0 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 15:09:29 service103 kernel: [] __down_write+0xb/0xd Jan 3 15:09:29 service103 kernel: [] down_write+0x11/0x13 Jan 3 15:09:29 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 15:09:29 service103 kernel: [] filter_destroy+0x99d/0x1fb0 [obdfilter] Jan 3 15:09:29 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] lh_read_lock+0x13/0x20 [obdclass] Jan 3 15:09:29 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] cfs_mem_cache_free+0x9/0x10 [libcfs] Jan 3 15:09:29 service103 kernel: [] ldlm_resource_putref_internal+0x3ab/0x460 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] ldlm_lock_put+0x372/0x3d0 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 15:09:29 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 15:09:29 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 15:09:30 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:09:30 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:09:30 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:09:30 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:09:30 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:09:30 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:09:30 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:09:30 service103 kernel: Jan 3 15:09:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632169.18413 Jan 3 15:10:11 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 15:10:14 service103 kernel: Lustre: 28243:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-182), not sending early reply Jan 3 15:10:14 service103 kernel: req@ffff810a58a53c00 x1388204553888134/t0 o6->17f54249-834c-7103-ed0e-40d91a2ec805@NET_0x500000a971a11_UUID:0/0 lens 512/400 e 1 to 0 dl 1325632219 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 15:10:14 service103 kernel: Lustre: 28243:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Jan 3 15:10:27 service103 kernel: Pid: 8034, comm: ll_ost_485 Jan 3 15:10:27 service103 kernel: Jan 3 15:10:27 service103 kernel: Call Trace: Jan 3 15:10:27 service103 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Jan 3 15:10:27 service103 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Jan 3 15:10:27 service103 kernel: [] __down+0xc3/0xd8 Jan 3 15:10:27 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:10:27 service103 kernel: [] __down_failed+0x35/0x3a Jan 3 15:10:27 service103 kernel: [] .text.lock.ldlm_resource+0x41/0x87 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:10:27 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] ldlm_lock_create+0xba/0x9f0 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] lustre_swab_reqbuf+0xfb/0x120 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Jan 3 15:10:27 service103 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 3 15:10:28 service103 kernel: [] ldlm_handle_enqueue+0x66f/0x1210 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Jan 3 15:10:28 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:10:28 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:10:28 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:10:28 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:10:28 service103 kernel: Jan 3 15:10:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632228.8034 Jan 3 15:12:09 service103 kernel: Lustre: 26513:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810a2912de00 already connecting Jan 3 15:12:09 service103 kernel: Lustre: 26513:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 1 previous similar message Jan 3 15:13:32 service103 kernel: Lustre: 9301:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810a393dfc00 already connecting Jan 3 15:13:32 service103 kernel: Lustre: 9301:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 2 previous similar messages Jan 3 15:13:46 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 5bdd42c7-c965-e756-f108-d718f4a68e6c (at (no nid)) in 170 seconds. I think it's dead, and I am evicting it. Jan 3 15:13:46 service103 kernel: Lustre: Skipped 12 previous similar messages Jan 3 15:13:49 service103 kernel: Lustre: Service thread pid 10625 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:13:49 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 15:13:49 service103 kernel: Pid: 10625, comm: ll_ost_435 Jan 3 15:13:49 service103 kernel: Jan 3 15:13:49 service103 kernel: Call Trace: Jan 3 15:13:49 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:13:49 service103 kernel: [] list_add+0xc/0xe Jan 3 15:13:49 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:13:49 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:13:49 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:13:49 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:13:49 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:13:49 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:13:49 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:13:49 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:13:49 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:13:49 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:13:49 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:13:49 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:13:49 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:13:49 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:13:49 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:13:49 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:13:50 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:13:50 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:13:50 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:13:50 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:13:50 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:13:50 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:13:50 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:13:50 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:13:50 service103 kernel: Jan 3 15:13:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632429.10625 Jan 3 15:13:57 service103 kernel: Pid: 6989, comm: ll_ost_472 Jan 3 15:13:57 service103 kernel: Jan 3 15:13:57 service103 kernel: Call Trace: Jan 3 15:13:57 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:13:57 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:13:57 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:13:57 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:13:57 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:13:57 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:13:57 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:13:57 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:13:57 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:13:57 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:13:57 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:13:57 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:13:57 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:13:57 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:13:57 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:13:58 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:13:58 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:13:58 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:13:58 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:13:58 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:13:58 service103 kernel: Jan 3 15:13:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632437.6989 Jan 3 15:13:59 service103 kernel: Pid: 9250, comm: ll_ost_75 Jan 3 15:13:59 service103 kernel: Jan 3 15:13:59 service103 kernel: Call Trace: Jan 3 15:13:59 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:13:59 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:13:59 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:13:59 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:13:59 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:13:59 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:13:59 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:13:59 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:13:59 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:13:59 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:13:59 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:13:59 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:13:59 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:13:59 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:13:59 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:13:59 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:13:59 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:13:59 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:14:00 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:14:00 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:14:00 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:14:00 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:14:00 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:14:00 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:14:00 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:14:00 service103 kernel: Jan 3 15:14:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632439.9250 Jan 3 15:14:00 service103 kernel: Pid: 9193, comm: ll_ost_18 Jan 3 15:14:00 service103 kernel: Jan 3 15:14:00 service103 kernel: Call Trace: Jan 3 15:14:01 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:14:01 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:14:01 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:14:01 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:14:01 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:14:01 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:14:01 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:14:01 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:14:01 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:14:01 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:14:01 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:14:01 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:14:02 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:14:02 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:14:02 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:14:02 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:14:02 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:14:02 service103 kernel: Jan 3 15:14:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632440.9193 Jan 3 15:14:37 service103 kernel: Lustre: 9195:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0062: exp ffff810a3e4d4a00 already connecting Jan 3 15:15:24 service103 kernel: Lustre: 1607:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: 17f54249-834c-7103-ed0e-40d91a2ec805 reconnecting Jan 3 15:15:24 service103 kernel: Lustre: 1607:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 533 previous similar messages Jan 3 15:16:25 service103 kernel: Pid: 7249, comm: ll_ost_418 Jan 3 15:16:25 service103 kernel: Jan 3 15:16:25 service103 kernel: Call Trace: Jan 3 15:16:25 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:16:25 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:16:25 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:16:25 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:16:25 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:16:25 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:16:25 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:16:25 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:16:25 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:16:25 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:16:25 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:16:25 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:16:25 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:16:25 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:16:25 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:16:26 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:16:26 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:16:26 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:16:26 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:16:26 service103 kernel: Jan 3 15:16:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632585.7249 Jan 3 15:16:28 service103 kernel: Lustre: 26524:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff8109bff08c00; still busy with 1 active RPCs Jan 3 15:16:28 service103 kernel: Lustre: 26524:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 554 previous similar messages Jan 3 15:16:30 service103 kernel: Pid: 7286, comm: ll_ost_421 Jan 3 15:16:30 service103 kernel: Jan 3 15:16:30 service103 kernel: Call Trace: Jan 3 15:16:30 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:16:30 service103 kernel: [] list_add+0xc/0xe Jan 3 15:16:31 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:16:31 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:16:31 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:16:31 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:16:31 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:16:31 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:16:31 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:16:31 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:16:31 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:16:31 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:16:31 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:16:31 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:16:31 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:16:31 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:16:31 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:16:32 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:16:32 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:16:32 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:16:32 service103 kernel: Jan 3 15:16:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632591.7286 Jan 3 15:16:33 service103 kernel: Pid: 9251, comm: ll_ost_76 Jan 3 15:16:33 service103 kernel: Jan 3 15:16:33 service103 kernel: Call Trace: Jan 3 15:16:33 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:16:33 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:16:33 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:16:33 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:16:33 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:16:33 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:16:33 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:16:33 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:16:33 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:16:33 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:16:33 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:16:33 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:16:33 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:16:33 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:16:33 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:16:34 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:16:34 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:16:34 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:16:34 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:16:34 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:16:34 service103 kernel: Jan 3 15:16:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632593.9251 Jan 3 15:16:35 service103 kernel: Pid: 16747, comm: ll_ost_463 Jan 3 15:16:35 service103 kernel: Jan 3 15:16:35 service103 kernel: Call Trace: Jan 3 15:16:35 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:16:35 service103 kernel: [] list_add+0xc/0xe Jan 3 15:16:35 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:16:35 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:16:35 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:16:35 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:16:35 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:16:35 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:16:35 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:16:35 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:16:35 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:16:35 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:16:35 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:16:35 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:16:35 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:16:35 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:16:35 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:16:35 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:16:35 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:16:36 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:16:36 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:16:36 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:16:36 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:16:36 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:16:36 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:16:36 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:16:36 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:16:36 service103 kernel: Jan 3 15:16:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632595.16747 Jan 3 15:16:36 service103 kernel: Pid: 6611, comm: ll_ost_266 Jan 3 15:16:37 service103 kernel: Jan 3 15:16:37 service103 kernel: Call Trace: Jan 3 15:16:37 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:16:37 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:16:37 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:16:37 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:16:37 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:16:37 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:16:37 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:16:37 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:16:37 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:16:37 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:16:37 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:16:37 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:16:38 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:16:38 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:16:38 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:16:38 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:16:38 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:16:39 service103 kernel: Jan 3 15:16:39 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632596.6611 Jan 3 15:17:12 service103 kernel: Lustre: Service thread pid 9325 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:17:12 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 15:17:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632632.9325 Jan 3 15:17:14 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632634.18415 Jan 3 15:17:14 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632634.18425 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.28166 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.28171 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.28228 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.9400 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.18377 Jan 3 15:17:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632635.28309 Jan 3 15:17:21 service103 kernel: Lustre: 30985:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff8109f5be4c00 already connecting Jan 3 15:17:21 service103 kernel: Lustre: 30985:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 4 previous similar messages Jan 3 15:18:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632688.26527 Jan 3 15:18:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632688.1612 Jan 3 15:18:09 service103 kernel: LustreError: 9182:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b96f57000 x1387481519696702/t0 o8->55292703-8b92-17f7-7381-401f6284429a@NET_0x500000a9719d3_UUID:0/0 lens 368/264 e 0 to 0 dl 1325632789 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 15:18:09 service103 kernel: LustreError: 9182:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 567 previous similar messages Jan 3 15:19:01 service103 kernel: Lustre: Service thread pid 25956 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:19:01 service103 kernel: Lustre: Skipped 10 previous similar messages Jan 3 15:19:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632741.25956 Jan 3 15:19:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632749.16752 Jan 3 15:19:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632751.9203 Jan 3 15:19:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632751.26507 Jan 3 15:19:49 service103 kernel: Lustre: 9227:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0062: exp ffff8109f9efb600 already connecting Jan 3 15:19:49 service103 kernel: Lustre: 9227:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 5 previous similar messages Jan 3 15:21:25 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 10.151.59.151@o2ib ns: filter-nbp6-OST0062_UUID lock: ffff8105c0f6d400/0x6ab269aac77aac0c lrc: 3/0,0 mode: PW/PW res: 278703/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0xecc6ce2d45087100 expref: 12 pid: 4513 timeout 6638468670 Jan 3 15:21:25 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 10.151.59.151@o2ib ns: filter-nbp6-OST0062_UUID lock: ffff81056a0a4a00/0x6ab269aac77aac91 lrc: 3/0,0 mode: PW/PW res: 278704/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0xecc6ce2d450878ee expref: 12 pid: 6603 timeout 6638468717 Jan 3 15:21:37 service103 kernel: Pid: 26503, comm: ll_ost_179 Jan 3 15:21:37 service103 kernel: Jan 3 15:21:37 service103 kernel: Call Trace: Jan 3 15:21:37 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:21:37 service103 kernel: [] list_add+0xc/0xe Jan 3 15:21:37 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:21:37 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:21:37 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:21:37 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:21:37 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:21:37 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:21:37 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:21:37 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:21:37 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:21:37 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:21:37 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:21:37 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:21:37 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:21:37 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:21:37 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:21:37 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:21:37 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:21:37 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:21:38 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:21:38 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:21:38 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:21:38 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:21:38 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:21:38 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:21:38 service103 kernel: Jan 3 15:21:38 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632897.26503 Jan 3 15:21:43 service103 kernel: Pid: 9246, comm: ll_ost_71 Jan 3 15:21:43 service103 kernel: Jan 3 15:21:43 service103 kernel: Call Trace: Jan 3 15:21:43 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:21:43 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:21:43 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:21:43 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:21:43 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:21:43 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:21:43 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:21:43 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:21:43 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:21:43 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:21:43 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:21:43 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:21:43 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:21:43 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:21:43 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:21:43 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:21:44 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:21:44 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:21:44 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:21:44 service103 kernel: Jan 3 15:21:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632903.9246 Jan 3 15:21:45 service103 kernel: Pid: 1597, comm: ll_ost_491 Jan 3 15:21:45 service103 kernel: Jan 3 15:21:45 service103 kernel: Call Trace: Jan 3 15:21:45 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:21:45 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:21:45 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:21:45 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:21:45 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:21:45 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:21:45 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:21:45 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:21:45 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:21:45 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:21:45 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:21:45 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:21:45 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:21:45 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:21:45 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:21:46 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:21:46 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:21:46 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:21:46 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:21:46 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:21:46 service103 kernel: Jan 3 15:21:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632905.1597 Jan 3 15:21:47 service103 kernel: Pid: 9226, comm: ll_ost_51 Jan 3 15:21:47 service103 kernel: Jan 3 15:21:47 service103 kernel: Call Trace: Jan 3 15:21:47 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:21:47 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:21:47 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:21:47 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:21:47 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:21:47 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:21:47 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:21:47 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:21:47 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:21:47 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:21:47 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:21:47 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:21:47 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:21:47 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:21:47 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:21:48 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:21:48 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:21:48 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:21:48 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:21:48 service103 kernel: Jan 3 15:21:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632907.9226 Jan 3 15:21:49 service103 kernel: Pid: 9290, comm: ll_ost_115 Jan 3 15:21:49 service103 kernel: Jan 3 15:21:49 service103 kernel: Call Trace: Jan 3 15:21:49 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:21:49 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:21:49 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:21:49 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:21:49 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:21:49 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:21:49 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:21:49 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:21:49 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:21:49 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:21:50 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:21:50 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:21:50 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:21:50 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:21:50 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:21:50 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:21:51 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:21:51 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:21:51 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:21:51 service103 kernel: Jan 3 15:21:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632908.9290 Jan 3 15:22:53 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fa21cf93-a978-6f00-557e-1048c45b07d4 (at (no nid)) in 176 seconds. I think it's dead, and I am evicting it. Jan 3 15:22:53 service103 kernel: Lustre: Skipped 21 previous similar messages Jan 3 15:23:08 service103 kernel: Lustre: Service thread pid 25952 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:23:08 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 15:23:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325632988.25952 Jan 3 15:23:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633015.16753 Jan 3 15:23:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633016.21332 Jan 3 15:23:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633020.1608 Jan 3 15:23:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633020.9230 Jan 3 15:23:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633022.24871 Jan 3 15:24:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633053.9222 Jan 3 15:24:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633061.6256 Jan 3 15:24:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633061.28968 Jan 3 15:24:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633063.29332 Jan 3 15:24:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633063.17986 Jan 3 15:24:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633080.29330 Jan 3 15:25:01 service103 kernel: Lustre: 26524:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0062: exp ffff8109d2ba8000 already connecting Jan 3 15:25:01 service103 kernel: Lustre: 26524:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 7 previous similar messages Jan 3 15:25:25 service103 kernel: Lustre: 8292:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: 61f2b20d-b871-39b2-363d-284f27370a17 reconnecting Jan 3 15:25:25 service103 kernel: Lustre: 8292:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 555 previous similar messages Jan 3 15:25:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633144.9277 Jan 3 15:25:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633145.8835 Jan 3 15:25:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633151.4470 Jan 3 15:26:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633176.21333 Jan 3 15:26:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633176.9245 Jan 3 15:26:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633178.9204 Jan 3 15:26:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633178.9414 Jan 3 15:26:30 service103 kernel: Lustre: 25960:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff8109bff08c00; still busy with 1 active RPCs Jan 3 15:26:30 service103 kernel: Lustre: 25960:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 567 previous similar messages Jan 3 15:26:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633194.17943 Jan 3 15:26:49 service103 kernel: Lustre: Service thread pid 9241 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:26:49 service103 kernel: Lustre: Skipped 13 previous similar messages Jan 3 15:26:49 service103 kernel: Pid: 9241, comm: ll_ost_66 Jan 3 15:26:49 service103 kernel: Jan 3 15:26:49 service103 kernel: Call Trace: Jan 3 15:26:49 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:26:49 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:26:49 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:26:49 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:26:49 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:26:49 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:26:49 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:26:49 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:26:49 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:26:49 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:26:49 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:26:49 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:26:49 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:26:49 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:26:49 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:26:49 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:26:49 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:26:49 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:26:50 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:26:50 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:26:50 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:26:50 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:26:50 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:26:50 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:26:50 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:26:50 service103 kernel: Jan 3 15:26:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633209.9241 Jan 3 15:26:55 service103 kernel: Pid: 9213, comm: ll_ost_38 Jan 3 15:26:55 service103 kernel: Jan 3 15:26:55 service103 kernel: Call Trace: Jan 3 15:26:55 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:26:55 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:26:55 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:26:55 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:26:55 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:26:55 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:26:55 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:26:55 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:26:55 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:26:55 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:26:55 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:26:55 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:26:55 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:26:55 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:26:55 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:26:55 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:26:56 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:26:56 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:26:56 service103 kernel: Jan 3 15:26:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633215.9213 Jan 3 15:26:57 service103 kernel: Pid: 9208, comm: ll_ost_33 Jan 3 15:26:57 service103 kernel: Jan 3 15:26:57 service103 kernel: Call Trace: Jan 3 15:26:57 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:26:57 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:26:57 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:26:57 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:26:57 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:26:57 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:26:57 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:26:57 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:26:57 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:26:57 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:26:57 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:26:57 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:26:57 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:26:57 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:26:57 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:26:57 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:26:57 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:26:57 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:26:57 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:26:58 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:26:58 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:26:58 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:26:58 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:26:58 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:26:58 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:26:58 service103 kernel: Jan 3 15:26:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633217.9208 Jan 3 15:26:58 service103 kernel: Pid: 8293, comm: ll_ost_293 Jan 3 15:26:58 service103 kernel: Jan 3 15:26:58 service103 kernel: Call Trace: Jan 3 15:26:58 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:26:58 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:26:59 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:26:59 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:26:59 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:26:59 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:26:59 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:26:59 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:26:59 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:26:59 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:26:59 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:26:59 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:26:59 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:26:59 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:26:59 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:26:59 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:27:00 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:27:00 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:27:00 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:27:00 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:27:00 service103 kernel: Jan 3 15:27:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633218.8293 Jan 3 15:27:00 service103 kernel: Pid: 3077, comm: ll_ost_204 Jan 3 15:27:00 service103 kernel: Jan 3 15:27:00 service103 kernel: Call Trace: Jan 3 15:27:01 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:27:01 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:27:01 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:27:01 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:27:01 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:27:01 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:27:01 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:27:01 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:27:01 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:27:01 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:27:01 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:27:01 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:27:01 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:27:02 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:27:02 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:27:02 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:27:02 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:27:02 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:27:02 service103 kernel: Jan 3 15:27:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633219.3077 Jan 3 15:27:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633219.9215 Jan 3 15:27:29 service103 kernel: Lustre: Service thread pid 4458 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:27:29 service103 kernel: Lustre: Skipped 20 previous similar messages Jan 3 15:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633249.24867 Jan 3 15:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633249.4458 Jan 3 15:28:09 service103 kernel: LustreError: 9216:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114) req@ffff81082d874400 x1386958894428808/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325633389 ref 1 fl Interpret:/0/0 rc -114/0 Jan 3 15:28:09 service103 kernel: LustreError: 9216:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 571 previous similar messages Jan 3 15:28:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633300.17953 Jan 3 15:28:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633309.5570 Jan 3 15:28:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633311.9235 Jan 3 15:28:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633312.2633 Jan 3 15:28:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633327.14519 Jan 3 15:28:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633328.4462 Jan 3 15:28:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633332.26502 Jan 3 15:28:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633332.9297 Jan 3 15:28:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633334.12793 Jan 3 15:29:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633373.28973 Jan 3 15:29:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633373.7290 Jan 3 15:29:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633375.9289 Jan 3 15:29:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633375.3079 Jan 3 15:29:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633392.29331 Jan 3 15:30:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633456.7206 Jan 3 15:30:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633457.30991 Jan 3 15:31:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633463.8336 Jan 3 15:31:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633483.17951 Jan 3 15:31:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633484.14528 Jan 3 15:31:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633506.6251 Jan 3 15:32:01 service103 kernel: Pid: 9183, comm: ll_ost_08 Jan 3 15:32:01 service103 kernel: Jan 3 15:32:01 service103 kernel: Call Trace: Jan 3 15:32:01 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:32:01 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:32:01 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:32:01 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:32:01 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:32:01 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:32:01 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:32:01 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:32:01 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:32:01 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:32:01 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:32:01 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:32:01 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:32:01 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:32:01 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:32:01 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:32:01 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:32:01 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:32:01 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:32:02 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:32:02 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:32:02 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:32:02 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:32:02 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:32:02 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:32:02 service103 kernel: Jan 3 15:32:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633521.9183 Jan 3 15:32:07 service103 kernel: Pid: 26506, comm: ll_ost_182 Jan 3 15:32:07 service103 kernel: Jan 3 15:32:07 service103 kernel: Call Trace: Jan 3 15:32:07 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:32:07 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:32:07 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:32:07 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:32:07 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:32:07 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:32:07 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:32:07 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:32:07 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:32:07 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:32:07 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:32:07 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:32:07 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:32:07 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:32:07 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:32:07 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:32:08 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:32:08 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:32:08 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:32:08 service103 kernel: Jan 3 15:32:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633527.26506 Jan 3 15:32:09 service103 kernel: Pid: 9239, comm: ll_ost_64 Jan 3 15:32:09 service103 kernel: Jan 3 15:32:09 service103 kernel: Call Trace: Jan 3 15:32:10 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:32:10 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:32:10 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:32:10 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:32:10 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:32:10 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:32:10 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:32:10 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:32:10 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:32:10 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:32:10 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:32:10 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:32:10 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:32:10 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:32:10 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:32:10 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:32:11 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:32:11 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:32:11 service103 kernel: Jan 3 15:32:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633530.9239 Jan 3 15:32:31 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.6.253@o2ib [old ver: 12, new ver: 12] Jan 3 15:32:32 service103 kernel: Pid: 8299, comm: ll_ost_299 Jan 3 15:32:32 service103 kernel: Jan 3 15:32:32 service103 kernel: Call Trace: Jan 3 15:32:32 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:32:32 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:32:32 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:32:32 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:32:32 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:32:32 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:32:32 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:32:32 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:32:32 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:32:32 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:32:32 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:32:32 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:32:32 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:32:32 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:32:32 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:32:32 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:32:32 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:32:32 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:32:33 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:32:33 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:32:33 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:32:33 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:32:33 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:32:33 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:32:33 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:32:33 service103 kernel: Jan 3 15:32:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633552.8299 Jan 3 15:32:33 service103 kernel: Pid: 6618, comm: ll_ost_273 Jan 3 15:32:33 service103 kernel: Jan 3 15:32:33 service103 kernel: Call Trace: Jan 3 15:32:33 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:32:34 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:32:34 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:32:34 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:32:34 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:32:34 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:32:34 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:32:34 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:32:34 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:32:34 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:32:34 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:32:34 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:32:34 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:32:34 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:32:34 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:32:35 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:32:35 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:32:35 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:32:35 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:32:35 service103 kernel: Jan 3 15:32:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633553.6618 Jan 3 15:32:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633554.1604 Jan 3 15:32:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633560.14523 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.30995 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.28974 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.28978 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.6250 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.1616 Jan 3 15:32:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633561.8281 Jan 3 15:32:47 service103 kernel: Lustre: 18410:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-182), not sending early reply Jan 3 15:32:47 service103 kernel: req@ffff8108178bd400 x1387572511430250/t0 o6->7abfce6f-1165-363a-a0e1-b95a1a28ea63@NET_0x500000a9719d4_UUID:0/0 lens 512/400 e 1 to 0 dl 1325633571 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 15:32:47 service103 kernel: Lustre: 18410:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages Jan 3 15:33:17 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fa21cf93-a978-6f00-557e-1048c45b07d4 (at (no nid)) in 176 seconds. I think it's dead, and I am evicting it. Jan 3 15:33:17 service103 kernel: Lustre: Skipped 48 previous similar messages Jan 3 15:33:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633621.7289 Jan 3 15:33:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633623.9231 Jan 3 15:33:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633624.9191 Jan 3 15:34:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633644.3176 Jan 3 15:34:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633645.7203 Jan 3 15:34:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633646.25958 Jan 3 15:34:22 service103 kernel: Lustre: 9176:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff8109f86e2600 already connecting Jan 3 15:34:22 service103 kernel: Lustre: 9176:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 26 previous similar messages Jan 3 15:34:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633685.9218 Jan 3 15:34:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633685.7287 Jan 3 15:34:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633687.24872 Jan 3 15:34:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633687.9240 Jan 3 15:35:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633703.6254 Jan 3 15:35:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633704.24874 Jan 3 15:35:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633705.15074 Jan 3 15:35:25 service103 kernel: Lustre: 9197:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: 7b75ecc3-0da4-d01e-9e9c-abe5754d562b reconnecting Jan 3 15:35:25 service103 kernel: Lustre: 9197:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 514 previous similar messages Jan 3 15:35:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633749.4468 Jan 3 15:35:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633751.9229 Jan 3 15:35:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633751.7285 Jan 3 15:36:08 service103 kernel: Lustre: Service thread pid 30989 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:36:08 service103 kernel: Lustre: Skipped 45 previous similar messages Jan 3 15:36:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633768.30989 Jan 3 15:36:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633769.8283 Jan 3 15:36:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633775.30993 Jan 3 15:36:30 service103 kernel: Lustre: 14526:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff810aefaf8e00; still busy with 1 active RPCs Jan 3 15:36:30 service103 kernel: Lustre: 14526:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 550 previous similar messages Jan 3 15:36:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633795.10632 Jan 3 15:36:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633796.4513 Jan 3 15:36:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633800.8290 Jan 3 15:36:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633801.6693 Jan 3 15:36:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633802.4457 Jan 3 15:36:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633818.26510 Jan 3 15:37:13 service103 kernel: Lustre: Service thread pid 9196 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:37:13 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 15:37:13 service103 kernel: Pid: 9196, comm: ll_ost_21 Jan 3 15:37:13 service103 kernel: Jan 3 15:37:13 service103 kernel: Call Trace: Jan 3 15:37:13 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:37:13 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:37:13 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:37:13 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:37:13 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:37:13 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:37:13 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:37:13 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:37:13 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:37:13 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:37:13 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:37:13 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:37:14 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:37:14 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:37:14 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:37:14 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:37:14 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:37:14 service103 kernel: Jan 3 15:37:15 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633833.9196 Jan 3 15:37:19 service103 kernel: Pid: 3090, comm: ll_ost_212 Jan 3 15:37:19 service103 kernel: Jan 3 15:37:19 service103 kernel: Call Trace: Jan 3 15:37:19 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:37:19 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:37:19 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:37:19 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:37:19 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:37:19 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:37:19 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:37:19 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:37:19 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:37:19 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:37:19 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:37:19 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:37:19 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:37:19 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:37:19 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:37:19 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:37:20 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:37:20 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:37:20 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:37:20 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:37:20 service103 kernel: Jan 3 15:37:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633839.3090 Jan 3 15:37:22 service103 kernel: Pid: 8278, comm: ll_ost_278 Jan 3 15:37:22 service103 kernel: Jan 3 15:37:22 service103 kernel: Call Trace: Jan 3 15:37:22 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:37:22 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:37:22 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:37:22 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:37:22 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:37:22 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:37:22 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:37:22 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:37:22 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:37:22 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:37:22 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:37:22 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:37:22 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:37:22 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:37:22 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:37:22 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:37:23 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:37:23 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:37:23 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:37:23 service103 kernel: Jan 3 15:37:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633842.8278 Jan 3 15:37:44 service103 kernel: Pid: 9194, comm: ll_ost_19 Jan 3 15:37:44 service103 kernel: Jan 3 15:37:44 service103 kernel: Call Trace: Jan 3 15:37:44 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:37:44 service103 kernel: [] list_add+0xc/0xe Jan 3 15:37:44 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:37:44 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:37:44 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:37:44 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:37:44 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:37:44 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:37:44 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:37:44 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:37:44 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:37:44 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:37:44 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:37:44 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:37:44 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:37:45 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:37:45 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:37:45 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:37:45 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:37:45 service103 kernel: Jan 3 15:37:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633864.9194 Jan 3 15:37:46 service103 kernel: Pid: 7291, comm: ll_ost_426 Jan 3 15:37:46 service103 kernel: Jan 3 15:37:46 service103 kernel: Call Trace: Jan 3 15:37:46 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:37:46 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:37:46 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:37:46 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:37:46 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:37:46 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:37:46 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:37:46 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:37:46 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:37:47 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:37:47 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:37:47 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:37:47 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:37:47 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:37:47 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:37:47 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:37:48 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:37:48 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:37:48 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:37:48 service103 kernel: Jan 3 15:37:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633865.7291 Jan 3 15:37:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633866.29924 Jan 3 15:37:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633872.18234 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.1601 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.9180 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.9233 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.1611 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.9299 Jan 3 15:37:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633873.10623 Jan 3 15:38:09 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.177@o2ib [old ver: 12, new ver: 12] Jan 3 15:38:10 service103 kernel: LustreError: 6257:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81036c1bc000 x1390004814962540/t0 o8->dc4d266f-a615-98e7-905d-ba4b335be73e@NET_0x500000a973585_UUID:0/0 lens 368/264 e 0 to 0 dl 1325633990 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 15:38:10 service103 kernel: LustreError: 6257:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 552 previous similar messages Jan 3 15:38:10 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.67@o2ib [old ver: 12, new ver: 12] Jan 3 15:38:10 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 2 previous similar messages Jan 3 15:38:15 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.42.147@o2ib [old ver: 12, new ver: 12] Jan 3 15:38:15 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 10 previous similar messages Jan 3 15:38:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633905.9214 Jan 3 15:38:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633911.4583 Jan 3 15:38:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633911.18383 Jan 3 15:38:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633921.24855 Jan 3 15:38:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633933.9298 Jan 3 15:38:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633935.9209 Jan 3 15:38:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633936.9296 Jan 3 15:39:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633951.28981 Jan 3 15:39:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633952.9223 Jan 3 15:39:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633984.28284 Jan 3 15:39:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633984.9430 Jan 3 15:39:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633984.18411 Jan 3 15:39:45 service103 kernel: Lustre: 9346:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 15:39:45 service103 kernel: req@ffff810b8f49f800 x1387391184926876/t0 o6->25f9e66c-f78b-0cea-78a2-0eb17ea2e34a@NET_0x500000a9719ce_UUID:0/0 lens 512/400 e 2 to 0 dl 1325633989 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 15:39:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633984.28298 Jan 3 15:39:45 service103 kernel: Lustre: 9346:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Jan 3 15:39:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633985.28294 Jan 3 15:39:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633985.9404 Jan 3 15:39:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633997.8297 Jan 3 15:39:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633998.17946 Jan 3 15:39:59 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633999.17949 Jan 3 15:39:59 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325633999.17987 Jan 3 15:40:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634018.6615 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634028.14533 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.24856 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.6617 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.6252 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.14520 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.6692 Jan 3 15:40:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634029.9227 Jan 3 15:41:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634081.9210 Jan 3 15:41:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634087.8298 Jan 3 15:41:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634089.6620 Jan 3 15:41:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634089.17944 Jan 3 15:41:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634089.4459 Jan 3 15:41:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634089.9197 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.28976 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.4279 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.29922 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.7207 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.9287 Jan 3 15:41:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634090.28977 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.9288 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.14526 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.9201 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.16624 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.1599 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.12129 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.25959 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.8284 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.29328 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.14518 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.4281 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.4510 Jan 3 15:41:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.1595 Jan 3 15:41:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634091.6601 Jan 3 15:41:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634092.6605 Jan 3 15:41:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634092.8280 Jan 3 15:41:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634092.14975 Jan 3 15:41:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634092.13458 Jan 3 15:41:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634095.12131 Jan 3 15:41:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634095.9184 Jan 3 15:41:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634096.9248 Jan 3 15:41:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634096.6257 Jan 3 15:41:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634107.25953 Jan 3 15:41:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634108.29329 Jan 3 15:41:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634112.9199 Jan 3 15:41:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634113.9228 Jan 3 15:41:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634114.6613 Jan 3 15:42:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634130.4464 Jan 3 15:42:34 service103 kernel: Pid: 9254, comm: ll_ost_79 Jan 3 15:42:34 service103 kernel: Jan 3 15:42:34 service103 kernel: Call Trace: Jan 3 15:42:34 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:42:34 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:42:34 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:42:34 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:42:34 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:42:34 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:42:34 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:42:34 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:42:34 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:42:34 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:42:34 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:42:34 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:42:34 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:42:34 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:42:34 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:42:34 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:42:35 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:42:35 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:42:35 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:42:35 service103 kernel: Jan 3 15:42:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634154.9254 Jan 3 15:43:41 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fa21cf93-a978-6f00-557e-1048c45b07d4 (at (no nid)) in 175 seconds. I think it's dead, and I am evicting it. Jan 3 15:43:41 service103 kernel: Lustre: Skipped 323 previous similar messages Jan 3 15:43:52 service103 kernel: Pid: 16746, comm: ll_ost_462 Jan 3 15:43:52 service103 kernel: Jan 3 15:43:52 service103 kernel: Call Trace: Jan 3 15:43:52 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:43:52 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:43:52 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:43:52 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:43:52 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:43:52 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:43:52 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:43:52 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:43:52 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:43:52 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:43:52 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:43:52 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:43:52 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:43:52 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:43:52 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:43:52 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:43:52 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:43:53 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:43:53 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:43:53 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:43:53 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:43:53 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:43:53 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:43:53 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:43:53 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:43:53 service103 kernel: Jan 3 15:43:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634232.16746 Jan 3 15:44:06 service103 kernel: Pid: 22784, comm: ll_ost_378 Jan 3 15:44:06 service103 kernel: Jan 3 15:44:06 service103 kernel: Call Trace: Jan 3 15:44:06 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:44:06 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:44:06 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:44:06 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:44:06 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:44:06 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:44:06 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:44:06 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:44:06 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:44:06 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:44:06 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:44:06 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:44:06 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:44:06 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:44:06 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:44:06 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:44:06 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:44:06 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:44:06 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:44:07 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:44:07 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:44:07 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:44:07 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:44:07 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:44:07 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:44:07 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:44:07 service103 kernel: Jan 3 15:44:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634246.22784 Jan 3 15:44:08 service103 kernel: Pid: 10631, comm: ll_ost_441 Jan 3 15:44:08 service103 kernel: Jan 3 15:44:08 service103 kernel: Call Trace: Jan 3 15:44:08 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:44:08 service103 kernel: [] list_add+0xc/0xe Jan 3 15:44:08 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:44:08 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:44:08 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:44:08 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:44:08 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:44:08 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:44:08 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:44:09 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:44:09 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:44:09 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:44:09 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:44:09 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:44:09 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:44:09 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:44:10 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:44:10 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:44:10 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:44:10 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:44:10 service103 kernel: Jan 3 15:44:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634247.10631 Jan 3 15:44:10 service103 kernel: Pid: 21331, comm: ll_ost_454 Jan 3 15:44:10 service103 kernel: Jan 3 15:44:10 service103 kernel: Call Trace: Jan 3 15:44:10 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:44:10 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:44:10 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:44:10 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:44:11 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:44:11 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:44:11 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:44:11 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:44:11 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:44:11 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:44:11 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:44:11 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:44:11 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:44:11 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:44:11 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:44:11 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:44:11 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:44:12 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:44:12 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:44:12 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:44:12 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 15:44:12 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:44:12 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:44:12 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:44:12 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:44:12 service103 kernel: Jan 3 15:44:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634248.21331 Jan 3 15:44:20 service103 kernel: Lustre: 18353:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-182), not sending early reply Jan 3 15:44:20 service103 kernel: req@ffff810613949450 x1389277663907429/t0 o6->1584acec-d396-3e73-847a-afc514e3b481@NET_0x500000a973645_UUID:0/0 lens 512/400 e 1 to 0 dl 1325634264 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 15:44:20 service103 kernel: Lustre: 18353:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jan 3 15:44:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634268.24861 Jan 3 15:44:29 service103 kernel: Lustre: 9181:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff8109f1ab2a00 already connecting Jan 3 15:44:29 service103 kernel: Lustre: 9181:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 85 previous similar messages Jan 3 15:44:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634269.6614 Jan 3 15:44:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634270.23457 Jan 3 15:44:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634271.26508 Jan 3 15:44:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634271.14529 Jan 3 15:44:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634271.21328 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.9178 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.6612 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.24873 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.4512 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.8286 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.1613 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.17948 Jan 3 15:44:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634272.23456 Jan 3 15:44:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634273.8295 Jan 3 15:44:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634273.1605 Jan 3 15:44:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634294.29325 Jan 3 15:44:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634294.9291 Jan 3 15:44:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634297.4472 Jan 3 15:45:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634310.30366 Jan 3 15:45:29 service103 kernel: Lustre: 12130:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: nbp6-mdtlov_UUID reconnecting Jan 3 15:45:29 service103 kernel: Lustre: 12130:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 549 previous similar messages Jan 3 15:45:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634331.7284 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.8888 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.6247 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.9243 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.24862 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.6992 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.6988 Jan 3 15:45:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634341.8294 Jan 3 15:46:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634361.28147 Jan 3 15:46:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634362.9401 Jan 3 15:46:28 service103 kernel: Lustre: Service thread pid 30998 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:46:28 service103 kernel: Lustre: Skipped 113 previous similar messages Jan 3 15:46:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634388.30998 Jan 3 15:46:32 service103 kernel: Lustre: 6991:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from nbp6-mdtlov_UUID@10.151.25.163@o2ib to 0xffff810aefaf8e00; still busy with 1 active RPCs Jan 3 15:46:32 service103 kernel: Lustre: 6991:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 533 previous similar messages Jan 3 15:46:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634393.4278 Jan 3 15:46:39 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634399.5574 Jan 3 15:46:59 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634419.24854 Jan 3 15:47:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634420.4456 Jan 3 15:47:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634424.4467 Jan 3 15:47:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634425.28975 Jan 3 15:47:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634426.10628 Jan 3 15:47:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634442.30994 Jan 3 15:47:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634446.9268 Jan 3 15:47:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634446.6607 Jan 3 15:47:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634446.6249 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.31039 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.3092 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.4466 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.9192 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.9295 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.25954 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.22785 Jan 3 15:47:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634447.9286 Jan 3 15:47:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634448.28966 Jan 3 15:47:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634448.3087 Jan 3 15:47:46 service103 kernel: Lustre: Service thread pid 30997 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:47:46 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 15:47:46 service103 kernel: Pid: 30997, comm: ll_ost_336 Jan 3 15:47:46 service103 kernel: Jan 3 15:47:46 service103 kernel: Call Trace: Jan 3 15:47:46 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:47:46 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:47:46 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:47:46 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:47:46 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:47:46 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:47:46 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:47:46 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:47:46 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:47:46 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:47:46 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:47:46 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:47:46 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:47:46 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:47:47 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:47:47 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:47:47 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:47:47 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:47:47 service103 kernel: Jan 3 15:47:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634466.30997 Jan 3 15:47:49 service103 kernel: Pid: 4280, comm: ll_ost_345 Jan 3 15:47:49 service103 kernel: Jan 3 15:47:49 service103 kernel: Call Trace: Jan 3 15:47:49 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:47:49 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:47:49 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:47:49 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:47:49 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:47:49 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:47:49 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:47:49 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:47:49 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:47:49 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:47:49 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:47:49 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:47:49 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:47:50 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:47:50 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:47:50 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:47:50 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:47:50 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:47:50 service103 kernel: Jan 3 15:47:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634469.4280 Jan 3 15:47:51 service103 kernel: Pid: 9175, comm: ll_ost_00 Jan 3 15:47:51 service103 kernel: Jan 3 15:47:51 service103 kernel: Call Trace: Jan 3 15:47:51 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:47:51 service103 kernel: [] list_add+0xc/0xe Jan 3 15:47:51 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:47:51 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:47:51 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:47:51 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:47:51 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:47:51 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:47:51 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:47:51 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:47:52 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:47:52 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:47:52 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:47:52 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:47:52 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:47:52 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:47:52 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:47:53 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:47:53 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:47:53 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:47:53 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:47:53 service103 kernel: Jan 3 15:47:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634470.9175 Jan 3 15:47:53 service103 kernel: Pid: 16839, comm: ll_ost_487 Jan 3 15:47:53 service103 kernel: Jan 3 15:47:53 service103 kernel: Call Trace: Jan 3 15:47:53 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:47:53 service103 kernel: [] list_add+0xc/0xe Jan 3 15:47:53 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:47:53 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:47:54 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:47:54 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:47:54 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:47:54 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:47:54 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:47:54 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:47:54 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:47:54 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:47:54 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:47:54 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:47:54 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:47:54 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:47:54 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:47:54 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:47:55 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:47:55 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:47:55 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:47:55 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:47:55 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:47:55 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:47:55 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:47:55 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:47:55 service103 kernel: Jan 3 15:47:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634472.16839 Jan 3 15:48:10 service103 kernel: LustreError: 26512:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8107c7694400 x1389011668564642/t0 o8->nbp6-mdtlov_UUID@:0/0 lens 368/264 e 0 to 0 dl 1325634590 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 15:48:10 service103 kernel: LustreError: 26512:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 616 previous similar messages Jan 3 15:48:17 service103 kernel: Pid: 9219, comm: ll_ost_44 Jan 3 15:48:17 service103 kernel: Jan 3 15:48:17 service103 kernel: Call Trace: Jan 3 15:48:17 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:48:17 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:48:17 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:48:17 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:48:17 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:48:17 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:48:17 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:48:17 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:48:17 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:48:17 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:48:17 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:48:17 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:48:17 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:48:17 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:48:17 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:48:18 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:48:18 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:48:18 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:48:18 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:48:18 service103 kernel: Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.9182 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.9283 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.17952 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.30367 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.1598 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.4469 Jan 3 15:48:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634497.9219 Jan 3 15:49:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634544.16838 Jan 3 15:49:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634558.6603 Jan 3 15:49:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634559.9271 Jan 3 15:49:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634560.1615 Jan 3 15:49:35 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634575.9279 Jan 3 15:49:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634576.6604 Jan 3 15:50:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634621.4454 Jan 3 15:50:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634621.9189 Jan 3 15:50:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634621.25960 Jan 3 15:50:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634621.13459 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.9202 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.8288 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.1594 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.28982 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.5572 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.10627 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.25957 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.9181 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.9269 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.17955 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.6990 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.7248 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.29333 Jan 3 15:50:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634622.24853 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.9206 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.22782 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.16750 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.10633 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.4465 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.9273 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.30985 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.25955 Jan 3 15:50:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634623.29923 Jan 3 15:50:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634643.9179 Jan 3 15:50:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634644.8276 Jan 3 15:50:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634644.14532 Jan 3 15:50:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634644.9186 Jan 3 15:50:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634645.19421 Jan 3 15:50:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634647.26513 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.26514 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.7244 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.9211 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.11604 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.9282 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.1602 Jan 3 15:50:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634653.30990 Jan 3 15:51:40 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634700.25951 Jan 3 15:51:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634705.24869 Jan 3 15:51:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634711.4460 Jan 3 15:52:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634736.3082 Jan 3 15:52:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634737.10626 Jan 3 15:52:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634738.9217 Jan 3 15:52:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634741.9224 Jan 3 15:52:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634741.21329 Jan 3 15:52:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634754.28970 Jan 3 15:52:58 service103 kernel: Pid: 9264, comm: ll_ost_89 Jan 3 15:52:58 service103 kernel: Jan 3 15:52:58 service103 kernel: Call Trace: Jan 3 15:52:58 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:52:58 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:52:58 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:52:58 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:52:58 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:52:58 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:52:58 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:52:58 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:52:58 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:52:58 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:52:58 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:52:58 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:52:58 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:52:58 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:52:58 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:52:58 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:52:58 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:52:58 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:52:59 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:52:59 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:52:59 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:52:59 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:52:59 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:52:59 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:52:59 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:52:59 service103 kernel: Jan 3 15:52:59 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634778.9264 Jan 3 15:53:16 service103 kernel: Pid: 3091, comm: ll_ost_213 Jan 3 15:53:16 service103 kernel: Jan 3 15:53:16 service103 kernel: Call Trace: Jan 3 15:53:16 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:53:16 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:53:16 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:53:16 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:53:16 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:53:16 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:53:16 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:53:16 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:53:16 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:53:16 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:53:16 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:53:16 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:53:16 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:53:16 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:53:16 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:53:16 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:53:16 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:53:16 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:53:16 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:53:17 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:53:17 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:53:17 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:53:17 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:53:17 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:53:17 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:53:17 service103 kernel: Jan 3 15:53:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634796.3091 Jan 3 15:53:17 service103 kernel: Pid: 7247, comm: ll_ost_416 Jan 3 15:53:17 service103 kernel: Jan 3 15:53:18 service103 kernel: Call Trace: Jan 3 15:53:18 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:53:18 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:53:18 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:53:18 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:53:18 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:53:18 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:53:18 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:53:18 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:53:18 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:53:18 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:53:18 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:53:19 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:53:19 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:53:19 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:53:19 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:53:19 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:53:19 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:53:20 service103 kernel: Jan 3 15:53:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634797.7247 Jan 3 15:53:20 service103 kernel: Pid: 1609, comm: ll_ost_503 Jan 3 15:53:20 service103 kernel: Jan 3 15:53:20 service103 kernel: Call Trace: Jan 3 15:53:20 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:53:20 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:53:20 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:53:20 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:53:20 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:53:20 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:53:20 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:53:20 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:53:21 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:53:21 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:53:21 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:53:21 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:53:21 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:53:21 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:53:21 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:53:21 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:53:22 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:53:22 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:53:22 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:53:22 service103 kernel: Jan 3 15:53:22 service103 kernel: Pid: 28979, comm: ll_ost_394 Jan 3 15:53:22 service103 kernel: Jan 3 15:53:22 service103 kernel: Call Trace: Jan 3 15:53:22 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:53:22 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:53:22 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:53:22 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:53:22 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:53:22 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:53:22 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:53:23 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:53:23 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:53:23 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:53:23 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:53:23 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:53:23 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:53:23 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:53:23 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:53:24 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:53:24 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:53:24 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:53:24 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:53:24 service103 kernel: Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.12132 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.9261 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.23618 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.4277 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.9302 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.9301 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.29327 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.17942 Jan 3 15:53:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.9234 Jan 3 15:53:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.28979 Jan 3 15:53:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634798.1609 Jan 3 15:53:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634800.28980 Jan 3 15:53:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634800.8287 Jan 3 15:53:39 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634819.26511 Jan 3 15:53:39 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634819.6619 Jan 3 15:53:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634822.9256 Jan 3 15:53:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634822.8887 Jan 3 15:54:05 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client bcb4dea8-923d-76f5-6860-589cd1a176b3 (at (no nid)) in 226 seconds. I think it's dead, and I am evicting it. Jan 3 15:54:05 service103 kernel: Lustre: Skipped 166 previous similar messages Jan 3 15:54:11 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.4.160@o2ib [old ver: 12, new ver: 12] Jan 3 15:54:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634856.30996 Jan 3 15:54:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634870.14524 Jan 3 15:54:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634871.9216 Jan 3 15:54:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634872.7288 Jan 3 15:54:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634888.6600 Jan 3 15:54:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634888.30982 Jan 3 15:54:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634892.9195 Jan 3 15:54:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634893.9232 Jan 3 15:55:11 service103 kernel: Lustre: 30988:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff8103bb4fae00 already connecting Jan 3 15:55:11 service103 kernel: Lustre: 30988:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 67 previous similar messages Jan 3 15:55:30 service103 kernel: Lustre: 9265:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: 198bf657-3565-0e9e-bc22-a2b75ee73a38 reconnecting Jan 3 15:55:30 service103 kernel: Lustre: 9265:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 552 previous similar messages Jan 3 15:55:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634934.4276 Jan 3 15:55:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634955.9244 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.9236 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.9190 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.3083 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.1607 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.9220 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.24865 Jan 3 15:56:05 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634965.16623 Jan 3 15:56:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634970.28176 Jan 3 15:56:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634971.5569 Jan 3 15:56:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634971.24858 Jan 3 15:56:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634971.15073 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.29326 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.19257 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.3075 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.1600 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.6248 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.24863 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.26509 Jan 3 15:56:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634972.22926 Jan 3 15:56:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634973.9253 Jan 3 15:56:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634973.26504 Jan 3 15:56:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634978.6606 Jan 3 15:56:32 service103 kernel: Lustre: 24864:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from b948bfca-ed5b-cdc5-3aca-c07ef12a8e6b@10.151.32.22@o2ib to 0xffff810a4467f000; still busy with 1 active RPCs Jan 3 15:56:32 service103 kernel: Lustre: 24864:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 545 previous similar messages Jan 3 15:56:34 service103 kernel: Lustre: Service thread pid 9188 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 15:56:34 service103 kernel: Lustre: Skipped 129 previous similar messages Jan 3 15:56:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634994.9188 Jan 3 15:56:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325634994.16749 Jan 3 15:56:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635012.9275 Jan 3 15:56:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635017.6602 Jan 3 15:57:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635023.26522 Jan 3 15:57:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635048.9242 Jan 3 15:57:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635049.9237 Jan 3 15:57:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635050.30983 Jan 3 15:57:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635051.22783 Jan 3 15:57:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635051.8285 Jan 3 15:57:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635066.26505 Jan 3 15:57:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635074.26782 Jan 3 15:57:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635074.14530 Jan 3 15:58:10 service103 kernel: Lustre: Service thread pid 9280 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 15:58:10 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 15:58:10 service103 kernel: Pid: 9280, comm: ll_ost_105 Jan 3 15:58:10 service103 kernel: Jan 3 15:58:10 service103 kernel: Call Trace: Jan 3 15:58:10 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:58:10 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:58:10 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:58:10 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:58:10 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:58:10 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:58:10 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:58:10 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:58:10 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:58:10 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:58:10 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:58:10 service103 kernel: [] lprocfs_counter_add+0x5b/0x100 [lvfs] Jan 3 15:58:10 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:58:10 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:58:10 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:58:11 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:58:11 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:58:11 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:58:11 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:58:11 service103 kernel: Jan 3 15:58:11 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635090.9280 Jan 3 15:58:11 service103 kernel: LustreError: 7205:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106672a9800 x1388403829997308/t0 o8->35e20ee4-1ef1-26f7-af2a-154df69cf230@NET_0x500000a97296c_UUID:0/0 lens 368/264 e 0 to 0 dl 1325635191 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 15:58:12 service103 kernel: LustreError: 7205:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 641 previous similar messages Jan 3 15:58:41 service103 kernel: Pid: 26525, comm: ll_ost_195 Jan 3 15:58:41 service103 kernel: Jan 3 15:58:41 service103 kernel: Call Trace: Jan 3 15:58:41 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:58:41 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:58:41 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:58:41 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:58:41 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:58:41 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:58:41 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:58:41 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:58:41 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:58:41 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:58:41 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:58:41 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:58:41 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:58:41 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:58:41 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:58:42 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:58:42 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:58:42 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:58:43 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:58:43 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:58:43 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:58:43 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:58:43 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:58:43 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:58:43 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:58:43 service103 kernel: Jan 3 15:58:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635121.26525 Jan 3 15:58:43 service103 kernel: Pid: 9259, comm: ll_ost_84 Jan 3 15:58:43 service103 kernel: Jan 3 15:58:43 service103 kernel: Call Trace: Jan 3 15:58:43 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:58:44 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:58:44 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:58:44 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:58:44 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:58:44 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:58:44 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:58:44 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:58:44 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:58:44 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:58:44 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:58:44 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:58:44 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:58:44 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:58:44 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:58:45 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 15:58:45 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:58:45 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:58:45 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:58:45 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:58:45 service103 kernel: Jan 3 15:58:45 service103 kernel: Pid: 9276, comm: ll_ost_101 Jan 3 15:58:45 service103 kernel: Jan 3 15:58:46 service103 kernel: Call Trace: Jan 3 15:58:46 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:58:46 service103 kernel: [] list_add+0xc/0xe Jan 3 15:58:46 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:58:46 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:58:46 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:58:46 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:58:46 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:58:46 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:58:46 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:58:46 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:58:46 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:58:46 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:58:46 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:58:47 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:58:47 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:58:47 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:58:47 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:58:47 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:58:47 service103 kernel: Jan 3 15:58:48 service103 kernel: Pid: 30992, comm: ll_ost_331 Jan 3 15:58:48 service103 kernel: Jan 3 15:58:48 service103 kernel: Call Trace: Jan 3 15:58:48 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 15:58:48 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 15:58:48 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 15:58:48 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 15:58:48 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 15:58:48 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 15:58:48 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 15:58:48 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 15:58:48 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 15:58:48 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 15:58:48 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 15:58:48 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 15:58:49 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 15:58:49 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 15:58:49 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] child_rip+0xa/0x11 Jan 3 15:58:49 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 15:58:49 service103 kernel: [] child_rip+0x0/0x11 Jan 3 15:58:49 service103 kernel: Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635122.8279 Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635122.6610 Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635122.11404 Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635123.30992 Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635123.9276 Jan 3 15:58:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635123.9259 Jan 3 15:59:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635146.1606 Jan 3 15:59:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635146.9200 Jan 3 15:59:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635146.6255 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.26524 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.10630 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.17954 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.9293 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.9262 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.9266 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.30368 Jan 3 15:59:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635147.9252 Jan 3 15:59:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635148.18235 Jan 3 15:59:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635148.8282 Jan 3 15:59:13 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635153.9247 Jan 3 15:59:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635168.9257 Jan 3 15:59:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635169.6609 Jan 3 15:59:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635169.9278 Jan 3 15:59:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635169.4516 Jan 3 15:59:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635170.9281 Jan 3 15:59:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635172.30988 Jan 3 15:59:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635172.4455 Jan 3 15:59:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635182.14522 Jan 3 15:59:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635183.30986 Jan 3 15:59:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635184.3078 Jan 3 16:00:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635200.9176 Jan 3 16:00:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635200.28967 Jan 3 16:00:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635246.28971 Jan 3 16:00:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635249.9274 Jan 3 16:01:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635266.9300 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.9207 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.8291 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.24870 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.9267 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.3080 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.28969 Jan 3 16:01:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635277.24864 Jan 3 16:01:41 service103 kernel: device ib1 entered promiscuous mode Jan 3 16:01:41 service103 kernel: device ib1 left promiscuous mode Jan 3 16:01:45 service103 kernel: device ib1 entered promiscuous mode Jan 3 16:01:58 service103 kernel: device ib1 left promiscuous mode Jan 3 16:02:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635321.29324 Jan 3 16:02:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635321.6608 Jan 3 16:02:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635321.26526 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.9212 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.17950 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.22781 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.29322 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.6616 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.26521 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.7246 Jan 3 16:02:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635322.9205 Jan 3 16:02:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635323.7243 Jan 3 16:02:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635323.9292 Jan 3 16:02:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635323.4461 Jan 3 16:02:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635323.9270 Jan 3 16:02:04 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635324.6991 Jan 3 16:02:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635328.29323 Jan 3 16:02:08 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635328.28972 Jan 3 16:02:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635329.16751 Jan 3 16:02:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635335.7205 Jan 3 16:02:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635344.9284 Jan 3 16:02:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635344.16748 Jan 3 16:02:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635347.14525 Jan 3 16:02:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635361.10635 Jan 3 16:02:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635361.3076 Jan 3 16:02:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635362.9177 Jan 3 16:02:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635378.26528 Jan 3 16:03:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635386.23619 Jan 3 16:03:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635386.30987 Jan 3 16:03:22 service103 kernel: Pid: 4517, comm: ll_ost_242 Jan 3 16:03:22 service103 kernel: Jan 3 16:03:22 service103 kernel: Call Trace: Jan 3 16:03:22 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:03:22 service103 kernel: [] list_add+0xc/0xe Jan 3 16:03:22 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:03:22 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:03:22 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:03:22 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:03:22 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:03:22 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:03:22 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:03:22 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:03:22 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:03:22 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:03:22 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:03:22 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:03:22 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:03:22 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:03:22 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:03:22 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:03:23 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:03:23 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 16:03:23 service103 kernel: [] smp_send_reschedule+0x4e/0x53 Jan 3 16:03:23 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:03:23 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:03:23 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:03:23 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:03:23 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:03:23 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:03:23 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:03:23 service103 kernel: Jan 3 16:03:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635402.4517 Jan 3 16:03:44 service103 kernel: Pid: 4463, comm: ll_ost_225 Jan 3 16:03:44 service103 kernel: Jan 3 16:03:44 service103 kernel: Call Trace: Jan 3 16:03:44 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:03:44 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:03:44 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:03:44 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:03:44 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:03:44 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:03:44 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:03:44 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:03:44 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:03:44 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:03:44 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:03:44 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:03:44 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:03:44 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:03:44 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:03:45 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:03:45 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:03:45 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:03:45 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:03:45 service103 kernel: Jan 3 16:03:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635424.4463 Jan 3 16:03:45 service103 kernel: Pid: 8292, comm: ll_ost_292 Jan 3 16:03:46 service103 kernel: Jan 3 16:03:46 service103 kernel: Call Trace: Jan 3 16:03:46 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:03:46 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:03:46 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:03:46 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:03:46 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:03:46 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:03:46 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:03:46 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:03:46 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:03:46 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:03:46 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:03:46 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:03:47 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:03:47 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:03:47 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:03:47 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:03:47 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:03:47 service103 kernel: Jan 3 16:03:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635425.8292 Jan 3 16:04:29 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fa21cf93-a978-6f00-557e-1048c45b07d4 (at (no nid)) in 175 seconds. I think it's dead, and I am evicting it. Jan 3 16:04:29 service103 kernel: Lustre: Skipped 137 previous similar messages Jan 3 16:04:40 service103 kernel: Pid: 26523, comm: ll_ost_193 Jan 3 16:04:40 service103 kernel: Jan 3 16:04:40 service103 kernel: Call Trace: Jan 3 16:04:40 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:04:40 service103 kernel: [] list_add+0xc/0xe Jan 3 16:04:40 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:04:40 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:04:40 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:04:40 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:04:40 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:04:40 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:04:40 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:04:40 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:04:40 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:04:40 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:04:40 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:04:40 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:04:40 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:04:40 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:04:41 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:04:41 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:04:41 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:04:41 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:04:41 service103 kernel: Jan 3 16:04:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635480.26523 Jan 3 16:04:54 service103 kernel: Pid: 7202, comm: ll_ost_347 Jan 3 16:04:54 service103 kernel: Jan 3 16:04:54 service103 kernel: Call Trace: Jan 3 16:04:54 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:04:54 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:04:54 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:04:54 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:04:54 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:04:54 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:04:54 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:04:54 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:04:54 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:04:54 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:04:54 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:04:54 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:04:54 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:04:54 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:04:54 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:04:55 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:04:55 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:04:55 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:04:55 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:04:55 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:04:55 service103 kernel: Jan 3 16:04:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635494.7202 Jan 3 16:04:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635496.1610 Jan 3 16:04:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635496.24857 Jan 3 16:04:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635496.9185 Jan 3 16:04:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635496.8834 Jan 3 16:04:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635496.9285 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.10629 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.12130 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.7204 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.30999 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.11405 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.24868 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.5575 Jan 3 16:04:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635497.9225 Jan 3 16:04:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635498.8296 Jan 3 16:04:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635498.9258 Jan 3 16:05:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635503.7292 Jan 3 16:05:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635512.9238 Jan 3 16:05:12 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635512.25725 Jan 3 16:05:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635517.5571 Jan 3 16:05:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635517.9255 Jan 3 16:05:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635518.26512 Jan 3 16:05:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635519.5578 Jan 3 16:05:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635519.1617 Jan 3 16:05:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635522.9198 Jan 3 16:05:33 service103 kernel: Lustre: 23212:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: nbp6-mdtlov_UUID reconnecting Jan 3 16:05:33 service103 kernel: Lustre: 23212:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 570 previous similar messages Jan 3 16:05:36 service103 kernel: Lustre: 7293:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810496dec200 already connecting Jan 3 16:05:36 service103 kernel: Lustre: 7293:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 93 previous similar messages Jan 3 16:05:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635558.24859 Jan 3 16:06:01 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635561.11605 Jan 3 16:06:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635579.19482 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.9221 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.8277 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.9272 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.4515 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.9263 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.3086 Jan 3 16:06:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635589.24860 Jan 3 16:06:39 service103 kernel: Lustre: Service thread pid 9265 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 16:06:39 service103 kernel: Lustre: Skipped 114 previous similar messages Jan 3 16:06:39 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635599.9265 Jan 3 16:07:16 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635636.4471 Jan 3 16:07:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635641.6599 Jan 3 16:07:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635648.5573 Jan 3 16:07:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635669.5577 Jan 3 16:07:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635671.14527 Jan 3 16:07:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635671.30984 Jan 3 16:07:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635671.8289 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.10634 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.7245 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.1596 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.9249 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.9187 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.1614 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.4514 Jan 3 16:07:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635672.14531 Jan 3 16:07:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635673.14521 Jan 3 16:07:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635673.1603 Jan 3 16:07:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635673.4453 Jan 3 16:07:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635673.6253 Jan 3 16:07:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635674.15735 Jan 3 16:07:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635678.30981 Jan 3 16:08:10 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635690.4511 Jan 3 16:08:14 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635694.17947 Jan 3 16:08:14 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635694.21330 Jan 3 16:08:17 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635697.17945 Jan 3 16:08:34 service103 kernel: Lustre: Service thread pid 9260 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 16:08:34 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 16:08:34 service103 kernel: Pid: 9260, comm: ll_ost_85 Jan 3 16:08:34 service103 kernel: Jan 3 16:08:34 service103 kernel: Call Trace: Jan 3 16:08:34 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:08:34 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:08:34 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:08:34 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:08:34 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:08:34 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:08:34 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:08:34 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:08:34 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:08:35 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:08:35 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:08:35 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:08:35 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:08:35 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:08:35 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:08:35 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:08:36 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:08:36 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:08:36 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:08:36 service103 kernel: Jan 3 16:08:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635714.9260 Jan 3 16:08:56 service103 kernel: Pid: 23212, comm: ll_ost_470 Jan 3 16:08:56 service103 kernel: Jan 3 16:08:56 service103 kernel: Call Trace: Jan 3 16:08:56 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:08:56 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:08:56 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:08:56 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:08:56 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:08:56 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:08:56 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:08:56 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:08:56 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:08:56 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:08:57 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:08:57 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:08:57 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:08:57 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:08:57 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:08:57 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:08:58 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:08:58 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:08:58 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:08:58 service103 kernel: Jan 3 16:08:58 service103 kernel: Pid: 7208, comm: ll_ost_353 Jan 3 16:08:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635737.23212 Jan 3 16:08:58 service103 kernel: Jan 3 16:08:58 service103 kernel: Call Trace: Jan 3 16:08:58 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:08:58 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:08:58 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:08:58 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:08:58 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:08:59 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:08:59 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:08:59 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:08:59 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:08:59 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:08:59 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:08:59 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:08:59 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:08:59 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:08:59 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:08:59 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:08:59 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:08:59 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:08:59 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:09:00 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:09:00 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:09:00 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:09:00 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:09:00 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:09:00 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:09:00 service103 kernel: Jan 3 16:09:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635737.7208 Jan 3 16:09:05 service103 kernel: Pid: 12792, comm: ll_ost_303 Jan 3 16:09:05 service103 kernel: Jan 3 16:09:05 service103 kernel: Call Trace: Jan 3 16:09:05 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:09:05 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:09:05 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:09:05 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:09:05 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:09:05 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:09:05 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:09:05 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:09:05 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:09:05 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:09:06 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:09:06 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:09:06 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:09:06 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:09:06 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:09:06 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:09:07 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:09:07 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:09:07 service103 kernel: Jan 3 16:09:07 service103 kernel: Pid: 4275, comm: ll_ost_340 Jan 3 16:09:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635745.12792 Jan 3 16:09:07 service103 kernel: Jan 3 16:09:07 service103 kernel: Call Trace: Jan 3 16:09:07 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 16:09:07 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 16:09:07 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 16:09:07 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 16:09:07 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 16:09:07 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 16:09:08 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 16:09:08 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 16:09:08 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 16:09:08 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 16:09:08 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 16:09:08 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 16:09:08 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 16:09:08 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 16:09:08 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 16:09:09 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 16:09:09 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 16:09:09 service103 kernel: [] child_rip+0xa/0x11 Jan 3 16:09:09 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 16:09:09 service103 kernel: [] child_rip+0x0/0x11 Jan 3 16:09:09 service103 kernel: Jan 3 16:09:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325635746.4275 Jan 3 16:10:11 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 16:15:53 service103 kernel: LustreError: 25201:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 607s Jan 3 16:15:53 service103 kernel: LustreError: 25201:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 608s Jan 3 16:17:30 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 972f96bb-44b2-0aad-2819-856831e72f57 (at 10.151.53.153@o2ib) in 151 seconds. I think it's dead, and I am evicting it. Jan 3 16:17:30 service103 kernel: Lustre: Skipped 65 previous similar messages Jan 3 16:17:40 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-115), not sending early reply Jan 3 16:17:40 service103 kernel: req@ffff810380582000 x1388403488622598/t0 o400->9b81a5e1-ed99-58f1-52da-9e847ccd000e@NET_0x500000a972bb7_UUID:0/0 lens 192/0 e 2 to 0 dl 1325636265 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:17:40 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jan 3 16:18:18 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-118), not sending early reply Jan 3 16:18:18 service103 kernel: req@ffff81080e988800 x1388958408080105/t0 o400->d9f4dd9c-3e82-17e4-7930-cf3109c6d374@NET_0x500000a970c98_UUID:0/0 lens 192/0 e 2 to 0 dl 1325636303 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:18:18 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 250153 previous similar messages Jan 3 16:18:53 service103 kernel: LustreError: 25741:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 788s Jan 3 16:19:34 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-184), not sending early reply Jan 3 16:19:34 service103 kernel: req@ffff8106c2c40000 x1388501261331192/t0 o400->063fb528-471e-28d6-1ea9-fad2b590d048@NET_0x500000a972845_UUID:0/0 lens 192/0 e 2 to 0 dl 1325636379 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:19:34 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 66599 previous similar messages Jan 3 16:21:53 service103 kernel: LustreError: 26347:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 968s Jan 3 16:21:53 service103 kernel: LustreError: 26347:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:23:06 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-321), not sending early reply Jan 3 16:23:06 service103 kernel: req@ffff810943897000 x1388501261332659/t0 o400->063fb528-471e-28d6-1ea9-fad2b590d048@NET_0x500000a972845_UUID:0/0 lens 192/0 e 2 to 0 dl 1325636591 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:23:06 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 536836 previous similar messages Jan 3 16:24:53 service103 kernel: LustreError: 26905:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 1148s Jan 3 16:24:53 service103 kernel: LustreError: 26905:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:27:39 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client 4c9ab2f5-e996-cf29-481b-2e244a808193 (at 10.151.32.83@o2ib) in 159 seconds. I think it's dead, and I am evicting it. Jan 3 16:27:39 service103 kernel: Lustre: Skipped 2390 previous similar messages Jan 3 16:27:53 service103 kernel: LustreError: 27510:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 1328s Jan 3 16:27:53 service103 kernel: LustreError: 27510:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:28:06 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply Jan 3 16:28:06 service103 kernel: req@ffff8105d59ca000 x1388670427356107/t0 o400->fe7f40e7-1345-c5d0-ee59-0af7dafffd3e@NET_0x500000a9716e2_UUID:0/0 lens 192/0 e 2 to 0 dl 1325636891 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:28:06 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 1945957 previous similar messages Jan 3 16:30:53 service103 kernel: LustreError: 28043:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 1508s Jan 3 16:30:53 service103 kernel: LustreError: 28043:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:33:53 service103 kernel: LustreError: 28746:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 1688s Jan 3 16:33:53 service103 kernel: LustreError: 28746:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:36:53 service103 kernel: LustreError: 29314:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 1868s Jan 3 16:36:53 service103 kernel: LustreError: 29314:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:37:58 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client b6c1430a-356e-e19d-43fc-3591b7f97447 (at 10.151.4.154@o2ib) in 157 seconds. I think it's dead, and I am evicting it. Jan 3 16:37:58 service103 kernel: Lustre: Skipped 4415 previous similar messages Jan 3 16:38:08 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-155), not sending early reply Jan 3 16:38:08 service103 kernel: req@ffff8103f2327800 x1387068320451305/t0 o400->52576dab-98ad-254d-00b9-1a7b928e5bb4@NET_0x500000a970bb5_UUID:0/0 lens 192/0 e 1 to 0 dl 1325637493 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:38:08 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 4184439 previous similar messages Jan 3 16:39:53 service103 kernel: LustreError: 29959:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 2048s Jan 3 16:39:53 service103 kernel: LustreError: 29959:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:42:53 service103 kernel: LustreError: 30501:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 2228s Jan 3 16:42:53 service103 kernel: LustreError: 30501:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 1 previous similar message Jan 3 16:48:13 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-155), not sending early reply Jan 3 16:48:13 service103 kernel: req@ffff8105682ce000 x1387753475583460/t0 o400->cdae5e50-9934-521e-617a-71f5627bce49@NET_0x500000a971a19_UUID:0/0 lens 192/0 e 1 to 0 dl 1325638098 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:48:13 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 1110 previous similar messages Jan 3 16:48:36 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client ca41e019-ccd8-2c01-689d-ec47d6b9b208 (at 10.151.42.26@o2ib) in 153 seconds. I think it's dead, and I am evicting it. Jan 3 16:48:36 service103 kernel: Lustre: Skipped 5020 previous similar messages Jan 3 16:48:53 service103 kernel: LustreError: 31653:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 2588s Jan 3 16:48:53 service103 kernel: LustreError: 31653:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 3 previous similar messages Jan 3 16:57:53 service103 kernel: LustreError: 1064:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 3128s Jan 3 16:57:53 service103 kernel: LustreError: 1064:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 5 previous similar messages Jan 3 16:58:38 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-155), not sending early reply Jan 3 16:58:38 service103 kernel: req@ffff8108fc1b0000 x1387753475674989/t0 o400->cdae5e50-9934-521e-617a-71f5627bce49@NET_0x500000a971a19_UUID:0/0 lens 192/0 e 1 to 0 dl 1325638723 ref 2 fl New:H/0/0 rc 0/0 Jan 3 16:58:38 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 454 previous similar messages Jan 3 16:58:56 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client 026b73bb-e31d-6d54-559d-b0183079bfbe (at 10.151.42.129@o2ib) in 154 seconds. I think it's dead, and I am evicting it. Jan 3 16:58:56 service103 kernel: Lustre: Skipped 839 previous similar messages Jan 3 17:04:28 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.5.95@o2ib [old ver: 12, new ver: 12] Jan 3 17:04:28 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.5.91@o2ib [old ver: 12, new ver: 12] Jan 3 17:08:39 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-155), not sending early reply Jan 3 17:08:39 service103 kernel: req@ffff810350872000 x1387753475771057/t0 o400->cdae5e50-9934-521e-617a-71f5627bce49@NET_0x500000a971a19_UUID:0/0 lens 192/0 e 1 to 0 dl 1325639323 ref 2 fl New:H/0/0 rc 0/0 Jan 3 17:08:39 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 335 previous similar messages Jan 3 17:09:23 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client f7d90588-e47f-946d-6e0d-de9eac9d14af (at 10.151.12.199@o2ib) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 17:09:23 service103 kernel: Lustre: Skipped 103 previous similar messages Jan 3 17:09:53 service103 kernel: LustreError: 3185:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 3848s Jan 3 17:09:53 service103 kernel: LustreError: 3185:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 7 previous similar messages Jan 3 17:10:11 service103 ntpd[24274]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 17:11:07 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.33.2@o2ib [old ver: 12, new ver: 12] Jan 3 17:11:07 service103 kernel: Lustre: 3238:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 17:18:39 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-155), not sending early reply Jan 3 17:18:39 service103 kernel: req@ffff81033cd20000 x1387753475861669/t0 o400->cdae5e50-9934-521e-617a-71f5627bce49@NET_0x500000a971a19_UUID:0/0 lens 192/0 e 1 to 0 dl 1325639923 ref 2 fl New:H/0/0 rc 0/0 Jan 3 17:18:39 service103 kernel: Lustre: 7293:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 335 previous similar messages Jan 3 17:19:48 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 88780a5f-a8cd-ffe4-26c3-aa879c500481 (at 10.151.22.231@o2ib) in 154 seconds. I think it's dead, and I am evicting it. Jan 3 17:19:48 service103 kernel: Lustre: Skipped 33161 previous similar messages Jan 3 17:21:53 service103 kernel: LustreError: 5551:0:(service.c:2124:ptlrpc_service_health_check()) ost: unhealthy - request has been waiting 4568s Jan 3 17:21:53 service103 kernel: LustreError: 5551:0:(service.c:2124:ptlrpc_service_health_check()) Skipped 7 previous similar messages Jan 3 17:53:45 service103 syslogd 1.4.1: restart. Jan 3 17:53:45 service103 kernel: klogd 1.4.1, log source = /proc/kmsg started. Jan 3 17:53:45 service103 kernel: Linux version 2.6.18-238.12.1.el5.20110722lustre186 (nobody@alcatraz) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Sun Jul 24 04:13:02 UTC 2011 Jan 3 17:53:45 service103 kernel: Command line: ro root=LABEL=sgiroot2 selinux=0 console=ttyS1,38400n8 crashkernel=128M@16M Jan 3 17:53:45 service103 kernel: BIOS-provided physical RAM map: Jan 3 17:53:45 service103 kernel: BIOS-e820: 0000000000010000 - 000000000009bc00 (usable) Jan 3 17:53:45 service103 kernel: BIOS-e820: 000000000009bc00 - 00000000000a0000 (reserved) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) Jan 3 17:53:45 service103 kernel: BIOS-e820: 0000000000100000 - 00000000bfef0000 (usable) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000bfef0000 - 00000000bff03000 (ACPI data) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000bff03000 - 00000000bff04000 (ACPI NVS) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000bff04000 - 00000000c0000000 (reserved) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) Jan 3 17:53:45 service103 portmap[4251]: user rpc not found, reverting to user bin Jan 3 17:53:45 service103 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) Jan 3 17:53:46 service103 rpc.statd[4308]: Version 1.0.9 Starting Jan 3 17:53:46 service103 kernel: BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved) Jan 3 17:53:46 service103 kernel: BIOS-e820: 0000000100000000 - 0000000c40000000 (usable) Jan 3 17:53:46 service103 kernel: DMI present. Jan 3 17:53:46 service103 run_srp_daemon[4349]: failed srp_daemon: [HCA=mlx4_0] [port=1] [exit status=1]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 17:53:46 service103 run_srp_daemon[4351]: failed srp_daemon: [HCA=mlx4_1] [port=1] [exit status=1]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 17:53:47 service103 kdump: kexec: loaded kdump kernel Jan 3 17:53:47 service103 kdump: started up Jan 3 17:53:47 service103 run_srp_daemon[4438]: starting srp_daemon: [HCA=mlx4_0] [port=1] Jan 3 17:53:47 service103 run_srp_daemon[4443]: starting srp_daemon: [HCA=mlx4_1] [port=1] Jan 3 17:53:48 service103 hcid[4469]: Bluetooth HCI daemon Jan 3 17:53:48 service103 sdpd[4473]: Bluetooth SDP daemon Jan 3 17:53:48 service103 hcid[4469]: Can't open system message bus connection: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory Jan 3 17:53:48 service103 hcid[4469]: Unable to get on D-Bus Jan 3 17:53:48 service103 kernel: No NUMA configuration found Jan 3 17:53:49 service103 kernel: Faking a node at 0000000000000000-0000000c40000000 Jan 3 17:53:49 service103 kernel: Bootmem setup node 0 0000000000000000-0000000c40000000 Jan 3 17:53:49 service103 kernel: ACPI: PM-Timer IO Port: 0x1008 Jan 3 17:53:49 service103 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jan 3 17:53:49 service103 kernel: Processor #0 7:7 APIC version 20 Jan 3 17:53:49 service103 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x04] enabled) Jan 3 17:53:49 service103 kernel: Processor #4 7:7 APIC version 20 Jan 3 17:53:49 service103 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) Jan 3 17:53:49 service103 kernel: Processor #1 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x05] enabled) Jan 3 17:53:50 service103 kernel: Processor #5 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled) Jan 3 17:53:50 service103 kernel: Processor #2 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x06] enabled) Jan 3 17:53:50 service103 kernel: Processor #6 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled) Jan 3 17:53:50 service103 kernel: Processor #3 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) Jan 3 17:53:50 service103 kernel: Processor #7 7:7 APIC version 20 Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) Jan 3 17:53:50 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1]) Jan 3 17:53:51 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1]) Jan 3 17:53:51 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1]) Jan 3 17:53:51 service103 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Jan 3 17:53:51 service103 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 Jan 3 17:53:51 service103 kernel: ACPI: IOAPIC (id[0x09] address[0xfec86000] gsi_base[24]) Jan 3 17:53:51 service103 kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec86000, GSI 24-47 Jan 3 17:53:51 service103 kernel: ACPI: IOAPIC (id[0x0a] address[0xfec89000] gsi_base[48]) Jan 3 17:53:51 service103 kernel: IOAPIC[2]: apic_id 10, version 32, address 0xfec89000, GSI 48-71 Jan 3 17:53:51 service103 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) Jan 3 17:53:51 service103 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jan 3 17:53:51 service103 kernel: Setting APIC routing to physical flat Jan 3 17:53:52 service103 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Jan 3 17:53:52 service103 kernel: Using ACPI (MADT) for SMP configuration information Jan 3 17:53:52 service103 kernel: Nosave address range: 000000000009b000 - 000000000009c000 Jan 3 17:53:52 service103 kernel: Nosave address range: 000000000009c000 - 00000000000a0000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000000a0000 - 00000000000e0000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000000e0000 - 0000000000100000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000bfef0000 - 00000000bff03000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000bff03000 - 00000000bff04000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000bff04000 - 00000000c0000000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000f0000000 - 00000000fec00000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000fec00000 - 00000000fec10000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000fec10000 - 00000000fee00000 Jan 3 17:53:52 service103 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000 Jan 3 17:53:53 service103 kernel: Nosave address range: 00000000fee01000 - 00000000ff000000 Jan 3 17:53:53 service103 kernel: Nosave address range: 00000000ff000000 - 0000000100000000 Jan 3 17:53:53 service103 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) Jan 3 17:53:53 service103 kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs Jan 3 17:53:53 service103 kernel: Built 1 zonelists. Total pages: 12405432 Jan 3 17:53:53 service103 kernel: Kernel command line: ro root=LABEL=sgiroot2 selinux=0 console=ttyS1,38400n8 crashkernel=128M@16M Jan 3 17:53:53 service103 kernel: Initializing CPU#0 Jan 3 17:53:53 service103 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Jan 3 17:53:53 service103 kernel: Console: colour VGA+ 80x25 Jan 3 17:53:53 service103 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Jan 3 17:53:53 service103 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Jan 3 17:53:53 service103 kernel: Checking aperture... Jan 3 17:53:53 service103 kernel: ACPI: DMAR not present Jan 3 17:53:53 service103 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Jan 3 17:53:54 service103 kernel: Placing software IO TLB between 0xf05e000 - 0x1305e000 Jan 3 17:53:54 service103 kernel: Memory: 49322844k/51380224k available (2665k kernel code, 1007248k reserved, 1746k data, 228k init) Jan 3 17:53:54 service103 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 5984.99 BogoMIPS (lpj=2992499) Jan 3 17:53:54 service103 kernel: kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved Jan 3 17:53:54 service103 kernel: Security Framework v1.0.0 initialized Jan 3 17:53:54 service103 kernel: SELinux: Disabled at boot. Jan 3 17:53:54 service103 kernel: Capability LSM initialized Jan 3 17:53:54 service103 kernel: Mount-cache hash table entries: 256 Jan 3 17:53:54 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:53:54 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:53:54 service103 kernel: using mwait in idle threads. Jan 3 17:53:55 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 17:53:55 service103 kernel: CPU: Processor Core ID: 0 Jan 3 17:53:55 service103 kernel: CPU0: Thermal monitoring enabled (TM2) Jan 3 17:53:55 service103 kernel: SMP alternatives: switching to UP code Jan 3 17:53:55 service103 kernel: ACPI: Core revision 20060707 Jan 3 17:53:55 service103 kernel: Using local APIC timer interrupts. Jan 3 17:53:55 service103 kernel: Detected 24.937 MHz APIC timer. Jan 3 17:53:55 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:53:55 service103 kernel: Booting processor 1/8 APIC 0x4 Jan 3 17:53:55 service103 kernel: Initializing CPU#1 Jan 3 17:53:55 service103 kernel: Calibrating delay using timer specific routine.. 5985.03 BogoMIPS (lpj=2992517) Jan 3 17:53:55 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:53:55 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:53:55 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 17:53:56 service103 kernel: CPU: Processor Core ID: 0 Jan 3 17:53:56 service103 kernel: CPU1: Thermal monitoring enabled (TM2) Jan 3 17:53:56 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:53:56 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:53:56 service103 kernel: Booting processor 2/8 APIC 0x1 Jan 3 17:53:56 service103 kernel: Initializing CPU#2 Jan 3 17:53:56 service103 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready. Jan 3 17:53:56 service103 kernel: Calibrating delay using timer specific routine.. 5985.00 BogoMIPS (lpj=2992503) Jan 3 17:53:56 service103 hidd[4659]: Bluetooth HID daemon Jan 3 17:53:56 service103 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead. Jan 3 17:53:57 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:53:57 service103 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s) Jan 3 17:53:57 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:53:57 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 17:53:57 service103 kernel: CPU: Processor Core ID: 1 Jan 3 17:53:58 service103 kernel: CPU2: Thermal monitoring enabled (TM2) Jan 3 17:53:58 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:53:58 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:53:58 service103 kernel: Booting processor 3/8 APIC 0x5 Jan 3 17:53:58 service103 kernel: Initializing CPU#3 Jan 3 17:53:58 service103 kernel: Calibrating delay using timer specific routine.. 5985.01 BogoMIPS (lpj=2992508) Jan 3 17:53:58 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:53:58 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:53:58 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 17:53:58 service103 kernel: CPU: Processor Core ID: 1 Jan 3 17:53:58 service103 kernel: CPU3: Thermal monitoring enabled (TM2) Jan 3 17:53:58 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:53:58 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:53:58 service103 kernel: Booting processor 4/8 APIC 0x2 Jan 3 17:53:59 service103 kernel: Initializing CPU#4 Jan 3 17:53:59 service103 kernel: Calibrating delay using timer specific routine.. 5984.89 BogoMIPS (lpj=2992445) Jan 3 17:53:59 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:53:59 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:53:59 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 17:53:59 service103 kernel: CPU: Processor Core ID: 2 Jan 3 17:53:59 service103 kernel: CPU4: Thermal monitoring enabled (TM2) Jan 3 17:53:59 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:53:59 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:53:59 service103 kernel: Booting processor 5/8 APIC 0x6 Jan 3 17:53:59 service103 kernel: Initializing CPU#5 Jan 3 17:53:59 service103 kernel: Calibrating delay using timer specific routine.. 5985.00 BogoMIPS (lpj=2992503) Jan 3 17:54:00 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:54:00 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:54:00 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 17:54:00 service103 kernel: CPU: Processor Core ID: 2 Jan 3 17:54:00 service103 kernel: CPU5: Thermal monitoring enabled (TM2) Jan 3 17:54:00 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:54:00 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:54:00 service103 kernel: Booting processor 6/8 APIC 0x3 Jan 3 17:54:00 service103 kernel: Initializing CPU#6 Jan 3 17:54:00 service103 kernel: Calibrating delay using timer specific routine.. 5984.99 BogoMIPS (lpj=2992497) Jan 3 17:54:00 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:54:00 service103 /etc/init.d/memlog[4779]: WARNING: Could not load module(s): worm Jan 3 17:54:00 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:54:01 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 17:54:01 service103 kernel: CPU: Processor Core ID: 3 Jan 3 17:54:01 service103 kernel: CPU6: Thermal monitoring enabled (TM2) Jan 3 17:54:01 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:54:01 service103 kernel: SMP alternatives: switching to SMP code Jan 3 17:54:01 service103 kernel: Booting processor 7/8 APIC 0x7 Jan 3 17:54:01 service103 kernel: Initializing CPU#7 Jan 3 17:54:01 service103 kernel: Calibrating delay using timer specific routine.. 5985.00 BogoMIPS (lpj=2992503) Jan 3 17:54:01 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 17:54:01 service103 kernel: CPU: L2 cache: 6144K Jan 3 17:54:01 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 17:54:01 service103 kernel: CPU: Processor Core ID: 3 Jan 3 17:54:01 service103 kernel: CPU7: Thermal monitoring enabled (TM2) Jan 3 17:54:01 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 17:54:02 service103 automount[4806]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master Jan 3 17:54:02 service103 kernel: Brought up 8 CPUs Jan 3 17:54:02 service103 nscd: 4824 Failed to run nscd as user 'nscd' Jan 3 17:54:03 service103 kernel: NMI watchdog testing PASSED. Jan 3 17:54:03 service103 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer. Jan 3 17:54:03 service103 kernel: time.c: Detected 2992.499 MHz processor. Jan 3 17:54:03 service103 kernel: migration_cost=12,9243 Jan 3 17:54:03 service103 kernel: checking if image is initramfs... it is Jan 3 17:54:03 service103 kernel: Freeing initrd memory: 2876k freed Jan 3 17:54:03 service103 kernel: NET: Registered protocol family 16 Jan 3 17:54:04 service103 kernel: ACPI: bus type pci registered Jan 3 17:54:04 service103 kernel: Warning: pci_mmcfg_init marking 256MB space uncacheable. Jan 3 17:54:04 service103 kernel: MCFG table requires 11MB uncacheable only. Try booting with acpi_mcfg_max_pci_bus_num=on Jan 3 17:54:04 service103 kernel: PCI: Using MMCONFIG at e0000000 Jan 3 17:54:04 service103 kernel: ACPI: Interpreter enabled Jan 3 17:54:04 service103 kernel: ACPI: Using IOAPIC for interrupt routing Jan 3 17:54:04 service103 kernel: ACPI: No dock devices found. Jan 3 17:54:04 service103 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Jan 3 17:54:04 service103 kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1 Jan 3 17:54:04 service103 kernel: PCI: Transparent bridge - 0000:00:1e.0 Jan 3 17:54:05 service103 hpiod: 1.6.7 accepting connections at 2208... Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *10 11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 *11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 7 10 11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 *11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *7 10 11 14 15) Jan 3 17:54:06 service103 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 6 7 10 11 14 15) *9 Jan 3 17:54:06 service103 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Jan 3 17:54:06 service103 kernel: pnp: PnP ACPI init Jan 3 17:54:06 service103 kernel: pnp: PnP ACPI: found 12 devices Jan 3 17:54:06 service103 kernel: usbcore: registered new driver usbfs Jan 3 17:54:07 service103 kernel: usbcore: registered new driver hub Jan 3 17:54:07 service103 kernel: PCI: Using ACPI for IRQ routing Jan 3 17:54:07 service103 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report Jan 3 17:54:07 service103 kernel: NetLabel: Initializing Jan 3 17:54:07 service103 kernel: NetLabel: domain hash size = 128 Jan 3 17:54:07 service103 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jan 3 17:54:07 service103 kernel: NetLabel: unlabeled traffic allowed by default Jan 3 17:54:08 service103 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0 Jan 3 17:54:08 service103 kernel: hpet0: 3 64-bit timers, 14318180 Hz Jan 3 17:54:08 service103 kernel: ACPI: DMAR not present Jan 3 17:54:08 service103 kernel: PCI-GART: No AMD northbridge found. Jan 3 17:54:08 service103 kernel: pnp: 00:01: iomem range 0xe0000000-0xefffffff could not be reserved Jan 3 17:54:08 service103 kernel: pnp: 00:01: iomem range 0xfee00000-0xfee0ffff could not be reserved Jan 3 17:54:08 service103 kernel: pnp: 00:01: iomem range 0xfec86000-0xfec86fff has been reserved Jan 3 17:54:09 service103 kernel: pnp: 00:01: iomem range 0xfec89000-0xfec89fff has been reserved Jan 3 17:54:09 service103 kernel: PCI: Bridge: 0000:00:01.0 Jan 3 17:54:09 service103 kernel: IO window: disabled. Jan 3 17:54:09 service103 OpenSM[4933]: Jan 3 17:54:09 service103 kernel: MEM window: d9200000-d92fffff Jan 3 17:54:09 service103 OpenSM[4933]: ONBOOT=no Jan 3 17:54:09 service103 kernel: PREFETCH window 0x00000000d8000000-0x00000000d87fffff Jan 3 17:54:09 service103 OpenSM[4933]: Loading Cached Option:guid = 0x0002c903000f9f83 Jan 3 17:54:09 service103 kernel: PCI: Bridge: 0000:00:03.0 Jan 3 17:54:09 service103 OpenSM[4933]: Loading Cached Option:honor_guid2lid_file = TRUE Jan 3 17:54:09 service103 kernel: IO window: 2000-2fff Jan 3 17:54:09 service103 OpenSM[4933]: Loading Cached Option:log_file = /var/log/opensm-mlx4_0_1.log Jan 3 17:54:09 service103 kernel: MEM window: d9300000-d93fffff Jan 3 17:54:10 service103 OpenSM[4933]: Loading Cached Option:dump_files_dir = /var/cache/opensm/mlx4_0_1 Jan 3 17:54:10 service103 kernel: PREFETCH window 0x00000000c2000000-0x00000000c21fffff Jan 3 17:54:10 service103 OpenSM[4935]: /var/log/opensm-mlx4_0_1.log log file opened Jan 3 17:54:10 service103 kernel: PCI: Bridge: 0000:00:05.0 Jan 3 17:54:10 service103 OpenSM[4935]: OpenSM 3.3.7 Jan 3 17:54:10 service103 kernel: IO window: disabled. Jan 3 17:54:10 service103 OpenSM[4935]: Entering DISCOVERING state Jan 3 17:54:10 service103 kernel: MEM window: d9400000-d94fffff Jan 3 17:54:10 service103 OpenSM[4935]: Entering MASTER state Jan 3 17:54:10 service103 kernel: PREFETCH window 0x00000000d8800000-0x00000000d8ffffff Jan 3 17:54:10 service103 OpenSM[4935]: SUBNET UP Jan 3 17:54:10 service103 kernel: PCI: Bridge: 0000:05:00.0 Jan 3 17:54:11 service103 kernel: IO window: disabled. Jan 3 17:54:11 service103 kernel: MEM window: disabled. Jan 3 17:54:11 service103 kernel: PREFETCH window: disabled. Jan 3 17:54:11 service103 kernel: PCI: Bridge: 0000:04:00.0 Jan 3 17:54:11 service103 OpenSM[4979]: Jan 3 17:54:11 service103 kernel: IO window: disabled. Jan 3 17:54:11 service103 OpenSM[4979]: ONBOOT=no Jan 3 17:54:11 service103 kernel: MEM window: disabled. Jan 3 17:54:11 service103 OpenSM[4979]: Loading Cached Option:guid = 0x0002c903000f9f8f Jan 3 17:54:11 service103 kernel: PREFETCH window: disabled. Jan 3 17:54:11 service103 OpenSM[4979]: Loading Cached Option:honor_guid2lid_file = TRUE Jan 3 17:54:11 service103 kernel: PCI: Bridge: 0000:04:00.3 Jan 3 17:54:11 service103 OpenSM[4979]: Loading Cached Option:log_file = /var/log/opensm-mlx4_1_1.log Jan 3 17:54:11 service103 kernel: IO window: disabled. Jan 3 17:54:11 service103 OpenSM[4979]: Loading Cached Option:dump_files_dir = /var/cache/opensm/mlx4_1_1 Jan 3 17:54:11 service103 kernel: MEM window: disabled. Jan 3 17:54:11 service103 OpenSM[4981]: /var/log/opensm-mlx4_1_1.log log file opened Jan 3 17:54:12 service103 kernel: PREFETCH window: disabled. Jan 3 17:54:12 service103 OpenSM[4981]: OpenSM 3.3.7 Jan 3 17:54:12 service103 kernel: PCI: Bridge: 0000:00:07.0 Jan 3 17:54:12 service103 OpenSM[4981]: Entering DISCOVERING state Jan 3 17:54:12 service103 kernel: IO window: disabled. Jan 3 17:54:12 service103 OpenSM[4981]: Entering MASTER state Jan 3 17:54:12 service103 kernel: MEM window: d9500000-d95fffff Jan 3 17:54:12 service103 OpenSM[4981]: SUBNET UP Jan 3 17:54:12 service103 kernel: PREFETCH window: disabled. Jan 3 17:54:12 service103 kernel: PCI: Bridge: 0000:00:09.0 Jan 3 17:54:12 service103 kernel: IO window: 3000-3fff Jan 3 17:54:12 service103 kernel: MEM window: d9600000-d96fffff Jan 3 17:54:12 service103 kernel: PREFETCH window 0x00000000c2200000-0x00000000c22fffff Jan 3 17:54:12 service103 kernel: PCI: Bridge: 0000:00:1c.0 Jan 3 17:54:12 service103 kernel: IO window: disabled. Jan 3 17:54:13 service103 kernel: MEM window: disabled. Jan 3 17:54:13 service103 kernel: PREFETCH window: disabled. Jan 3 17:54:13 service103 boot.booted: TEMPO:service103 EVENT:NODE_BOOTED APP:BOOT.BOOTED DATE:Jan 3 2012 17:54:13 VERSION:1.0 TEXT:Node booted successfully. Jan 3 17:54:13 service103 kernel: PCI: Bridge: 0000:00:1e.0 Jan 3 17:54:13 service103 kernel: IO window: 4000-4fff Jan 3 17:54:13 service103 multipathd: sdb: add path (uevent) Jan 3 17:54:13 service103 kernel: MEM window: d9700000-d97fffff Jan 3 17:54:13 service103 kernel: PREFETCH window 0x00000000d0000000-0x00000000d7ffffff Jan 3 17:54:14 service103 kernel: GSI 16 sharing vector 0xA9 and IRQ 16 Jan 3 17:54:14 service103 kernel: ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 48 (level, low) -> IRQ 169 Jan 3 17:54:14 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:16 10] Jan 3 17:54:14 service103 logger: Adjusted blockdev Jan 3 17:54:14 service103 logger: Adjusted sdb max_sectors_kb=4096 Jan 3 17:54:14 service103 kernel: GSI 17 sharing vector 0xB1 and IRQ 17 Jan 3 17:54:14 service103 logger: Adjusted sdb scheduler=deadline Jan 3 17:54:14 service103 kernel: ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 50 (level, low) -> IRQ 177 Jan 3 17:54:14 service103 logger: Adjusted blockdev Jan 3 17:54:14 service103 logger: Adjusted blockdev Jan 3 17:54:15 service103 logger: Adjected sdb timeout=280 Jan 3 17:54:15 service103 logger: Adjusted sdd max_sectors_kb=4096 Jan 3 17:54:15 service103 logger: Adjusted sdd scheduler=deadline Jan 3 17:54:15 service103 multipathd: ddn6a-nbp6-ost2: event checker started Jan 3 17:54:15 service103 kernel: GSI 18 sharing vector 0xB9 and IRQ 18 Jan 3 17:54:15 service103 logger: Adjected sdd timeout=280 Jan 3 17:54:15 service103 logger: Adjusted sdc max_sectors_kb=4096 Jan 3 17:54:15 service103 logger: Adjusted sdc scheduler=deadline Jan 3 17:54:15 service103 multipathd: sdc: add path (uevent) Jan 3 17:54:15 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:32 10] Jan 3 17:54:15 service103 multipathd: ddn6a-nbp6-ost10: event checker started Jan 3 17:54:15 service103 logger: Adjected sdc timeout=280 Jan 3 17:54:15 service103 kernel: ACPI: PCI Interrupt 0000:00:05.0[A] -> GSI 52 (level, low) -> IRQ 185 Jan 3 17:54:15 service103 multipathd: sdd: add path (uevent) Jan 3 17:54:16 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:48 10] Jan 3 17:54:16 service103 kernel: GSI 19 sharing vector 0xC1 and IRQ 19 Jan 3 17:54:16 service103 logger: Adjusted blockdev Jan 3 17:54:16 service103 multipathd: ddn6a-nbp6-ost18: event checker started Jan 3 17:54:16 service103 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 17:54:16 service103 logger: Adjusted sde max_sectors_kb=4096 Jan 3 17:54:16 service103 multipathd: sde: add path (uevent) Jan 3 17:54:16 service103 logger: Adjusted sde scheduler=deadline Jan 3 17:54:16 service103 logger: Adjusted blockdev Jan 3 17:54:16 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:64 10] Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 kernel: ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 17:54:17 service103 logger: Adjected sde timeout=280 Jan 3 17:54:17 service103 logger: Adjusted sdf max_sectors_kb=4096 Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 last message repeated 2 times Jan 3 17:54:17 service103 multipathd: ddn6a-nbp6-ost26: event checker started Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 last message repeated 2 times Jan 3 17:54:17 service103 logger: Adjusted sdg max_sectors_kb=4096 Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 last message repeated 3 times Jan 3 17:54:17 service103 logger: Adjusted sdh max_sectors_kb=4096 Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:17 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 kernel: ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 17:54:17 service103 logger: Adjusted sdj max_sectors_kb=4096 Jan 3 17:54:17 service103 logger: Adjusted sdf scheduler=deadline Jan 3 17:54:17 service103 logger: Adjusted sdi max_sectors_kb=4096 Jan 3 17:54:17 service103 logger: Adjusted sdl max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 multipathd: dm-0: add map (uevent) Jan 3 17:54:18 service103 logger: Adjusted sdq max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdm max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdk max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdg scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdt max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdp max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdo max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 last message repeated 2 times Jan 3 17:54:18 service103 logger: Adjusted sds max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdu max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdh scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 logger: Adjusted sdr max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 last message repeated 3 times Jan 3 17:54:18 service103 logger: Adjusted sdj scheduler=deadline Jan 3 17:54:18 service103 multipathd: dm-0: devmap already registered Jan 3 17:54:18 service103 logger: Adjected sdf timeout=280 Jan 3 17:54:18 service103 logger: Adjusted sdi scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdl scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdv max_sectors_kb=4096 Jan 3 17:54:18 service103 logger: Adjusted sdq scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdm scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdk scheduler=deadline Jan 3 17:54:18 service103 logger: Adjected sdg timeout=280 Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 logger: Adjusted sdt scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted blockdev Jan 3 17:54:18 service103 logger: Adjusted sdp scheduler=deadline Jan 3 17:54:18 service103 logger: Adjusted sdo scheduler=deadline Jan 3 17:54:19 service103 logger: Adjusted sdx max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdw max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdy max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sds scheduler=deadline Jan 3 17:54:19 service103 logger: Adjusted sdu scheduler=deadline Jan 3 17:54:19 service103 logger: Adjected sdh timeout=280 Jan 3 17:54:19 service103 logger: Adjusted sdz max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdr scheduler=deadline Jan 3 17:54:19 service103 logger: Adjusted sdab max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdn max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjected sdj timeout=280 Jan 3 17:54:19 service103 logger: Adjusted sdaa max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdac max_sectors_kb=4096 Jan 3 17:54:19 service103 multipathd: sdf: add path (uevent) Jan 3 17:54:19 service103 logger: Adjected sdi timeout=280 Jan 3 17:54:19 service103 logger: Adjected sdl timeout=280 Jan 3 17:54:19 service103 logger: Adjusted sdv scheduler=deadline Jan 3 17:54:19 service103 logger: Adjected sdq timeout=280 Jan 3 17:54:19 service103 logger: Adjected sdm timeout=280 Jan 3 17:54:19 service103 logger: Adjected sdk timeout=280 Jan 3 17:54:19 service103 logger: Adjected sdt timeout=280 Jan 3 17:54:19 service103 logger: Adjusted sdad max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjusted sdae max_sectors_kb=4096 Jan 3 17:54:19 service103 logger: Adjected sdp timeout=280 Jan 3 17:54:19 service103 logger: Adjected sdo timeout=280 Jan 3 17:54:19 service103 logger: Adjusted sdx scheduler=deadline Jan 3 17:54:19 service103 logger: Adjusted sdw scheduler=deadline Jan 3 17:54:20 service103 logger: Adjusted sdy scheduler=deadline Jan 3 17:54:20 service103 logger: Adjected sds timeout=280 Jan 3 17:54:20 service103 logger: Adjected sdu timeout=280 Jan 3 17:54:20 service103 logger: Adjusted sdz scheduler=deadline Jan 3 17:54:20 service103 logger: Adjected sdr timeout=280 Jan 3 17:54:20 service103 logger: Adjusted sdab scheduler=deadline Jan 3 17:54:20 service103 kernel: GSI 20 sharing vector 0xC9 and IRQ 20 Jan 3 17:54:20 service103 logger: Adjusted sdn scheduler=deadline Jan 3 17:54:20 service103 logger: Adjusted sdaa scheduler=deadline Jan 3 17:54:20 service103 logger: Adjusted sdac scheduler=deadline Jan 3 17:54:20 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:80 10] Jan 3 17:54:20 service103 logger: Adjected sdv timeout=280 Jan 3 17:54:20 service103 logger: Adjusted sdad scheduler=deadline Jan 3 17:54:20 service103 logger: Adjusted sdae scheduler=deadline Jan 3 17:54:21 service103 logger: Adjected sdx timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdw timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdy timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdz timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdab timeout=280 Jan 3 17:54:21 service103 kernel: ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 56 (level, low) -> IRQ 201 Jan 3 17:54:21 service103 logger: Adjected sdn timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdaa timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdac timeout=280 Jan 3 17:54:21 service103 multipathd: ddn6a-nbp6-ost34: event checker started Jan 3 17:54:21 service103 logger: Adjected sdad timeout=280 Jan 3 17:54:21 service103 logger: Adjected sdae timeout=280 Jan 3 17:54:22 service103 multipathd: sdg: add path (uevent) Jan 3 17:54:22 service103 kernel: GSI 21 sharing vector 0xD1 and IRQ 21 Jan 3 17:54:22 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:96 10] Jan 3 17:54:22 service103 kernel: ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 209 Jan 3 17:54:22 service103 multipathd: ddn6a-nbp6-ost42: event checker started Jan 3 17:54:22 service103 multipathd: dm-1: add map (uevent) Jan 3 17:54:23 service103 multipathd: dm-1: devmap already registered Jan 3 17:54:23 service103 kernel: NET: Registered protocol family 2 Jan 3 17:54:23 service103 multipathd: sdi: add path (uevent) Jan 3 17:54:23 service103 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Jan 3 17:54:23 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:128 10 round-robin 0 1 1 8:16 10] Jan 3 17:54:23 service103 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Jan 3 17:54:23 service103 multipathd: sdh: add path (uevent) Jan 3 17:54:23 service103 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jan 3 17:54:23 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:112 10] Jan 3 17:54:23 service103 kernel: TCP: Hash tables configured (established 262144 bind 65536) Jan 3 17:54:23 service103 multipathd: ddn6a-nbp6-ost50: event checker started Jan 3 17:54:23 service103 kernel: TCP reno registered Jan 3 17:54:23 service103 multipathd: sdj: add path (uevent) Jan 3 17:54:23 service103 kernel: Simple Boot Flag at 0x41 set to 0x80 Jan 3 17:54:23 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:144 10] Jan 3 17:54:23 service103 kernel: audit: initializing netlink socket (disabled) Jan 3 17:54:23 service103 multipathd: ddn6a-nbp6-ost58: event checker started Jan 3 17:54:23 service103 kernel: type=2000 audit(1325613161.589:1): initialized Jan 3 17:54:24 service103 multipathd: sdk: add path (uevent) Jan 3 17:54:24 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:160 10 round-robin 0 1 1 8:32 10] Jan 3 17:54:24 service103 multipathd: dm-2: add map (uevent) Jan 3 17:54:24 service103 multipathd: dm-2: devmap already registered Jan 3 17:54:24 service103 multipathd: sdl: add path (uevent) Jan 3 17:54:24 service103 kernel: Total HugeTLB memory allocated, 0 Jan 3 17:54:24 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:176 10] Jan 3 17:54:24 service103 kernel: VFS: Disk quotas dquot_6.5.1 Jan 3 17:54:24 service103 multipathd: ddn6a-nbp6-ost66: event checker started Jan 3 17:54:24 service103 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jan 3 17:54:24 service103 multipathd: sdm: add path (uevent) Jan 3 17:54:24 service103 kernel: Initializing Cryptographic API Jan 3 17:54:24 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:192 10] Jan 3 17:54:24 service103 kernel: alg: No test for crc32c (crc32c-generic) Jan 3 17:54:24 service103 multipathd: ddn6a-nbp6-ost74: event checker started Jan 3 17:54:24 service103 kernel: ksign: Installing public key data Jan 3 17:54:24 service103 multipathd: sdn: add path (uevent) Jan 3 17:54:24 service103 kernel: Loading keyring Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:208 10] Jan 3 17:54:25 service103 kernel: io scheduler noop registered Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost82: event checker started Jan 3 17:54:25 service103 kernel: io scheduler anticipatory registered Jan 3 17:54:25 service103 multipathd: sdo: add path (uevent) Jan 3 17:54:25 service103 kernel: io scheduler deadline registered Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 8:224 10 round-robin 0 1 1 8:48 10] Jan 3 17:54:25 service103 kernel: io scheduler cfq registered (default) Jan 3 17:54:25 service103 multipathd: sdp: add path (uevent) Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:240 10] Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost90: event checker started Jan 3 17:54:25 service103 multipathd: sdq: add path (uevent) Jan 3 17:54:25 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:0 10 round-robin 0 1 1 8:64 10] Jan 3 17:54:25 service103 multipathd: dm-3: add map (uevent) Jan 3 17:54:26 service103 multipathd: dm-3: devmap already registered Jan 3 17:54:26 service103 multipathd: sdr: add path (uevent) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:16 10] Jan 3 17:54:26 service103 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU0] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost98: event checker started Jan 3 17:54:26 service103 multipathd: sds: add path (uevent) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:32 10 round-robin 0 1 1 8:80 10] Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU1] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: sdt: add path (uevent) Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU2] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:48 10] Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU3] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost106: event checker started Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU4] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: sdu: add path (uevent) Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU5] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 65:64 10] Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU6] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: ddn6a-nbp6-ost114: event checker started Jan 3 17:54:26 service103 kernel: ACPI: Processor [CPU7] (supports 8 throttling states) Jan 3 17:54:26 service103 multipathd: dm-4: add map (uevent) Jan 3 17:54:26 service103 kernel: Real Time Clock Driver v1.12ac Jan 3 17:54:26 service103 multipathd: dm-4: devmap already registered Jan 3 17:54:27 service103 xinetd[5211]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. Jan 3 17:54:27 service103 multipathd: sdv: add path (uevent) Jan 3 17:54:27 service103 kernel: Non-volatile memory driver v1.2 Jan 3 17:54:27 service103 xinetd[5211]: Started working: 2 available services Jan 3 17:54:27 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:80 10 round-robin 0 1 1 8:96 10] Jan 3 17:54:27 service103 kernel: Linux agpgart interface v0.101 (c) Dave Jones Jan 3 17:54:27 service103 multipathd: dm-5: add map (uevent) Jan 3 17:54:27 service103 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Jan 3 17:54:27 service103 multipathd: dm-5: devmap already registered Jan 3 17:54:27 service103 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 3 17:54:27 service103 multipathd: sdw: add path (uevent) Jan 3 17:54:27 service103 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jan 3 17:54:27 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:96 10 round-robin 0 1 1 8:112 10] Jan 3 17:54:27 service103 kernel: 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 3 17:54:27 service103 multipathd: dm-0: add map (uevent) Jan 3 17:54:27 service103 kernel: 00:0b: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jan 3 17:54:27 service103 multipathd: dm-0: devmap already registered Jan 3 17:54:27 service103 kernel: brd: module loaded Jan 3 17:54:27 service103 multipathd: sdx: add path (uevent) Jan 3 17:54:28 service103 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Jan 3 17:54:28 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:112 10 round-robin 0 1 1 8:144 10] Jan 3 17:54:28 service103 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Jan 3 17:54:28 service103 multipathd: dm-6: add map (uevent) Jan 3 17:54:28 service103 kernel: ESB2: IDE controller at PCI slot 0000:00:1f.1 Jan 3 17:54:28 service103 multipathd: dm-6: devmap already registered Jan 3 17:54:28 service103 kernel: ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 16 (level, low) -> IRQ 209 Jan 3 17:54:28 service103 multipathd: sdy: add path (uevent) Jan 3 17:54:28 service103 kernel: ESB2: chipset revision 9 Jan 3 17:54:28 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:128 10 round-robin 0 1 1 8:176 10] Jan 3 17:54:28 service103 kernel: ESB2: not 100% native mode: will probe irqs later Jan 3 17:54:28 service103 multipathd: dm-7: add map (uevent) Jan 3 17:54:28 service103 kernel: ide0: BM-DMA at 0x1860-0x1867, BIOS settings: hda:pio, hdb:DMA Jan 3 17:54:28 service103 multipathd: dm-7: devmap already registered Jan 3 17:54:28 service103 multipathd: sdz: add path (uevent) Jan 3 17:54:28 service103 kernel: hdb: MATSHITADVD-RAM UJ870PC, ATAPI CD/DVD-ROM drive Jan 3 17:54:28 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:144 10 round-robin 0 1 1 8:192 10] Jan 3 17:54:28 service103 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 Jan 3 17:54:29 service103 multipathd: dm-1: add map (uevent) Jan 3 17:54:29 service103 multipathd: dm-1: devmap already registered Jan 3 17:54:29 service103 kernel: ide-floppy driver 0.99.newide Jan 3 17:54:29 service103 multipathd: sdaa: add path (uevent) Jan 3 17:54:29 service103 kernel: usbcore: registered new driver hiddev Jan 3 17:54:29 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:160 10 round-robin 0 1 1 8:208 10] Jan 3 17:54:29 service103 kernel: usbcore: registered new driver usbhid Jan 3 17:54:29 service103 multipathd: dm-8: add map (uevent) Jan 3 17:54:29 service103 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Jan 3 17:54:29 service103 multipathd: dm-8: devmap already registered Jan 3 17:54:29 service103 kernel: PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12 Jan 3 17:54:29 service103 multipathd: sdab: add path (uevent) Jan 3 17:54:29 service103 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jan 3 17:54:29 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:176 10 round-robin 0 1 1 8:240 10] Jan 3 17:54:29 service103 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jan 3 17:54:29 service103 multipathd: dm-9: add map (uevent) Jan 3 17:54:29 service103 kernel: mice: PS/2 mouse device common for all mice Jan 3 17:54:29 service103 multipathd: dm-9: devmap already registered Jan 3 17:54:30 service103 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 Jan 3 17:54:30 service103 multipathd: sdac: add path (uevent) Jan 3 17:54:30 service103 kernel: md: bitmap version 4.39 Jan 3 17:54:30 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:192 10 round-robin 0 1 1 65:16 10] Jan 3 17:54:30 service103 kernel: TCP bic registered Jan 3 17:54:30 service103 multipathd: sdad: add path (uevent) Jan 3 17:54:30 service103 kernel: Initializing IPsec netlink socket Jan 3 17:54:30 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:208 10 round-robin 0 1 1 65:48 10] Jan 3 17:54:30 service103 kernel: NET: Registered protocol family 1 Jan 3 17:54:30 service103 multipathd: dm-10: add map (uevent) Jan 3 17:54:30 service103 kernel: NET: Registered protocol family 17 Jan 3 17:54:30 service103 multipathd: dm-10: devmap already registered Jan 3 17:54:30 service103 kernel: ACPI: (supports S0 S1 S4 S5) Jan 3 17:54:30 service103 multipathd: sdae: add path (uevent) Jan 3 17:54:30 service103 kernel: Initalizing network drop monitor service Jan 3 17:54:30 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:224 10 round-robin 0 1 1 65:64 10] Jan 3 17:54:30 service103 kernel: Freeing unused kernel memory: 228k freed Jan 3 17:54:30 service103 multipathd: dm-2: add map (uevent) Jan 3 17:54:31 service103 kernel: Write protecting the kernel read-only data: 600k Jan 3 17:54:31 service103 multipathd: dm-2: devmap already registered Jan 3 17:54:31 service103 kernel: SCSI subsystem initialized Jan 3 17:54:31 service103 multipathd: dm-11: add map (uevent) Jan 3 17:54:31 service103 multipathd: dm-11: devmap already registered Jan 3 17:54:31 service103 kernel: GSI 22 sharing vector 0x5A and IRQ 22 Jan 3 17:54:31 service103 multipathd: dm-3: add map (uevent) Jan 3 17:54:31 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 90 Jan 3 17:54:31 service103 multipathd: dm-3: devmap already registered Jan 3 17:54:31 service103 multipathd: dm-12: add map (uevent) Jan 3 17:54:31 service103 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller Jan 3 17:54:31 service103 multipathd: dm-12: devmap already registered Jan 3 17:54:31 service103 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1 Jan 3 17:54:31 service103 multipathd: dm-4: add map (uevent) Jan 3 17:54:31 service103 kernel: ehci_hcd 0000:00:1d.7: debug port 1 Jan 3 17:54:31 service103 multipathd: dm-4: devmap already registered Jan 3 17:54:31 service103 multipathd: dm-13: add map (uevent) Jan 3 17:54:32 service103 kernel: ehci_hcd 0000:00:1d.7: irq 90, io mem 0xd9804000 Jan 3 17:54:32 service103 multipathd: dm-13: devmap already registered Jan 3 17:54:32 service103 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Jan 3 17:54:32 service103 multipathd: dm-14: add map (uevent) Jan 3 17:54:32 service103 kernel: usb usb1: configuration #1 chosen from 1 choice Jan 3 17:54:32 service103 multipathd: dm-14: devmap already registered Jan 3 17:54:32 service103 kernel: hub 1-0:1.0: USB hub found Jan 3 17:54:32 service103 multipathd: dm-5: add map (uevent) Jan 3 17:54:32 service103 kernel: hub 1-0:1.0: 6 ports detected Jan 3 17:54:32 service103 multipathd: dm-5: devmap already registered Jan 3 17:54:32 service103 multipathd: dm-6: add map (uevent) Jan 3 17:54:32 service103 kernel: USB Universal Host Controller Interface driver v3.0 Jan 3 17:54:32 service103 multipathd: dm-6: devmap already registered Jan 3 17:54:32 service103 kernel: GSI 23 sharing vector 0x62 and IRQ 23 Jan 3 17:54:32 service103 multipathd: dm-7: add map (uevent) Jan 3 17:54:32 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 20 (level, low) -> IRQ 98 Jan 3 17:54:32 service103 multipathd: dm-7: devmap already registered Jan 3 17:54:33 service103 multipathd: dm-8: add map (uevent) Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.0: irq 98, io base 0x00001800 Jan 3 17:54:33 service103 kernel: usb usb2: configuration #1 chosen from 1 choice Jan 3 17:54:33 service103 kernel: hub 2-0:1.0: USB hub found Jan 3 17:54:33 service103 kernel: hub 2-0:1.0: 2 ports detected Jan 3 17:54:33 service103 kernel: usb 1-6: new high speed USB device using ehci_hcd and address 2 Jan 3 17:54:33 service103 kernel: GSI 24 sharing vector 0x6A and IRQ 24 Jan 3 17:54:33 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 21 (level, low) -> IRQ 106 Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3 Jan 3 17:54:33 service103 kernel: uhci_hcd 0000:00:1d.1: irq 106, io base 0x00001820 Jan 3 17:54:33 service103 kernel: usb usb3: configuration #1 chosen from 1 choice Jan 3 17:54:33 service103 kernel: hub 3-0:1.0: USB hub found Jan 3 17:54:33 service103 kernel: hub 3-0:1.0: 2 ports detected Jan 3 17:54:34 service103 kernel: usb 1-6: configuration #1 chosen from 1 choice Jan 3 17:54:34 service103 kernel: input: Peppercon AG Multidevice as /class/input/input0 Jan 3 17:54:34 service103 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1d.7-6 Jan 3 17:54:34 service103 kernel: input: Peppercon AG Multidevice as /class/input/input1 Jan 3 17:54:34 service103 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1d.7-6 Jan 3 17:54:34 service103 kernel: GSI 25 sharing vector 0x72 and IRQ 25 Jan 3 17:54:34 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 22 (level, low) -> IRQ 114 Jan 3 17:54:34 service103 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller Jan 3 17:54:35 service103 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4 Jan 3 17:54:35 service103 kernel: uhci_hcd 0000:00:1d.2: irq 114, io base 0x00001840 Jan 3 17:54:35 service103 kernel: usb usb4: configuration #1 chosen from 1 choice Jan 3 17:54:35 service103 kernel: hub 4-0:1.0: USB hub found Jan 3 17:54:35 service103 kernel: hub 4-0:1.0: 2 ports detected Jan 3 17:54:35 service103 kernel: Fusion MPT base driver 3.04.15rh Jan 3 17:54:35 service103 kernel: Copyright (c) 1999-2008 LSI Corporation Jan 3 17:54:35 service103 kernel: Fusion MPT SAS Host driver 3.04.15rh Jan 3 17:54:35 service103 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 50 (level, low) -> IRQ 177 Jan 3 17:54:35 service103 kernel: mptbase: ioc0: Initiating bringup Jan 3 17:54:35 service103 kernel: ioc0: LSISAS1068E B3: Capabilities={Initiator} Jan 3 17:54:35 service103 kernel: scsi0 : ioc0: LSISAS1068E B3, FwRev=01170400h, Ports=1, MaxQ=286, IRQ=177 Jan 3 17:54:35 service103 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 5, phy 0, sas_addr 0x3f2e56397eaa8476 Jan 3 17:54:35 service103 kernel: Vendor: ATA Model: HDS725050KLA360 Rev: AD1A Jan 3 17:54:36 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:36 service103 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 2, phy 1, sas_addr 0x3f2e563b87908979 Jan 3 17:54:36 service103 kernel: Vendor: ATA Model: HDS725050KLA360 Rev: AD1A Jan 3 17:54:36 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:36 service103 kernel: mptsas: ioc0: attaching raid volume, channel 1, id 0 Jan 3 17:54:36 service103 kernel: Vendor: LSILOGIC Model: Logical Volume Rev: 3000 Jan 3 17:54:36 service103 kernel: Type: Direct-Access ANSI SCSI revision: 02 Jan 3 17:54:36 service103 kernel: SCSI device sda: 976482304 512-byte hdwr sectors (499959 MB) Jan 3 17:54:36 service103 kernel: sda: Write Protect is off Jan 3 17:54:37 service103 kernel: SCSI device sda: drive cache: write through Jan 3 17:54:37 service103 kernel: SCSI device sda: 976482304 512-byte hdwr sectors (499959 MB) Jan 3 17:54:37 service103 kernel: sda: Write Protect is off Jan 3 17:54:37 service103 kernel: SCSI device sda: drive cache: write through Jan 3 17:54:37 service103 kernel: sda: sda1 sda2 sda3 < sda5 sda6 sda7 sda8 > Jan 3 17:54:37 service103 kernel: sd 0:1:0:0: Attached scsi disk sda Jan 3 17:54:37 service103 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Jan 3 17:54:37 service103 kernel: device-mapper: uevent: version 1.0.3 Jan 3 17:54:37 service103 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com Jan 3 17:54:37 service103 kernel: device-mapper: dm-raid45: initialized v0.2594l Jan 3 17:54:37 service103 kernel: Fusion MPT FC Host driver 3.04.15rh Jan 3 17:54:37 service103 kernel: Fusion MPT misc device (ioctl) driver 3.04.15rh Jan 3 17:54:38 service103 kernel: mptctl: Registered with Fusion MPT base driver Jan 3 17:54:38 service103 kernel: mptctl: /dev/mptctl @ (major,minor=10,220) Jan 3 17:54:38 service103 kernel: BIOS EDD facility v0.16 2004-Jun-25, 1 devices found Jan 3 17:54:38 service103 kernel: megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006) Jan 3 17:54:38 service103 kernel: megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006) Jan 3 17:54:38 service103 kernel: megasas: 00.00.04.31-RH1 Tues. June. 15 14:13:02 EST 2010 Jan 3 17:54:38 service103 kernel: 802.1Q VLAN Support v1.8 Ben Greear Jan 3 17:54:38 service103 kernel: All bugs added by David S. Miller Jan 3 17:54:38 service103 kernel: kjournald starting. Commit interval 5 seconds Jan 3 17:54:38 service103 kernel: EXT3-fs: mounted filesystem with ordered data mode. Jan 3 17:54:38 service103 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 0 Jan 3 17:54:38 service103 kernel: scsi 0:0:1:0: Attached scsi generic sg1 type 0 Jan 3 17:54:38 service103 kernel: sd 0:1:0:0: Attached scsi generic sg2 type 0 Jan 3 17:54:38 service103 kernel: input: PC Speaker as /class/input/input2 Jan 3 17:54:39 service103 kernel: intel_rng: FWH not detected Jan 3 17:54:39 service103 kernel: dca service started, version 1.8 Jan 3 17:54:39 service103 kernel: memtrack::init_module done. Jan 3 17:54:39 service103 kernel: EDAC MC: Ver: 2.0.1 Jul 24 2011 Jan 3 17:54:39 service103 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011) Jan 3 17:54:39 service103 kernel: mlx4_core: Initializing 0000:01:00.0 Jan 3 17:54:39 service103 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 48 (level, low) -> IRQ 169 Jan 3 17:54:39 service103 kernel: hdb: ATAPI 24X DVD-ROM DVD-R-RAM CD-R/RW drive, 2048kB Cache, UDMA(33) Jan 3 17:54:39 service103 kernel: Uniform CD-ROM driver Revision: 3.20 Jan 3 17:54:39 service103 kernel: Initializing USB Mass Storage driver... Jan 3 17:54:39 service103 kernel: scsi1 : SCSI emulation for USB Mass Storage devices Jan 3 17:54:40 service103 kernel: usbcore: registered new driver usb-storage Jan 3 17:54:40 service103 kernel: USB Mass Storage support registered. Jan 3 17:54:40 service103 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2-1 Jan 3 17:54:40 service103 kernel: Copyright (c) 2007-2009 Intel Corporation. Jan 3 17:54:40 service103 kernel: EDAC MC0: Giving out device to i5400_edac.c I5400: DEV 0000:00:10.0 Jan 3 17:54:40 service103 kernel: GSI 26 sharing vector 0xC2 and IRQ 26 Jan 3 17:54:40 service103 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 194 Jan 3 17:54:40 service103 ntpdate[6309]: step time server 172.29.0.1 offset 0.025662 sec Jan 3 17:54:40 service103 kernel: ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 56 (level, low) -> IRQ 201 Jan 3 17:54:40 service103 ntpd[6988]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) Jan 3 17:54:40 service103 kernel: igb 0000:08:00.0: Disabling ASPM L0s upstream switch port 0000:00:09.0 Jan 3 17:54:40 service103 ntpd[6989]: precision = 1.000 usec Jan 3 17:54:41 service103 kernel: mlx4_core: Initializing 0000:03:00.0 Jan 3 17:54:41 service103 ntpd[6989]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jan 3 17:54:41 service103 kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 52 (level, low) -> IRQ 185 Jan 3 17:54:41 service103 ntpd[6989]: Listening on interface wildcard, ::#123 Disabled Jan 3 17:54:41 service103 kernel: igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection Jan 3 17:54:41 service103 ntpd[6989]: Listening on interface lo, ::1#123 Enabled Jan 3 17:54:42 service103 kernel: igb 0000:08:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:30:48:c4:4f:0c Jan 3 17:54:42 service103 ntpd[6989]: Listening on interface ib1, fe80::202:c903:f:9f84#123 Enabled Jan 3 17:54:42 service103 kernel: igb 0000:08:00.0: eth0: PBA No: ffffff-0ff Jan 3 17:54:42 service103 ntpd[6989]: Listening on interface eth0, fe80::230:48ff:fec4:4f0c#123 Enabled Jan 3 17:54:42 service103 kernel: igb 0000:08:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Jan 3 17:54:42 service103 kernel: GSI 27 sharing vector 0x3B and IRQ 27 Jan 3 17:54:42 service103 kernel: ACPI: PCI Interrupt 0000:08:00.1[B] -> GSI 70 (level, low) -> IRQ 59 Jan 3 17:54:42 service103 ntpd[6989]: Listening on interface ib0, fe80::202:c903:f:9f83#123 Enabled Jan 3 17:54:42 service103 kernel: igb 0000:08:00.1: Disabling ASPM L0s upstream switch port 0000:00:09.0 Jan 3 17:54:42 service103 ntpd[6989]: Listening on interface lo, 127.0.0.1#123 Enabled Jan 3 17:54:43 service103 ntpd[6989]: Listening on interface eth0, 172.29.1.8#123 Enabled Jan 3 17:54:43 service103 kernel: igb 0000:08:00.1: Intel(R) Gigabit Ethernet Network Connection Jan 3 17:54:43 service103 ntpd[6989]: Listening on interface ib0, 10.150.25.157#123 Enabled Jan 3 17:54:43 service103 kernel: igb 0000:08:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:30:48:c4:4f:0d Jan 3 17:54:43 service103 ntpd[6989]: Listening on interface ib1, 10.151.25.157#123 Enabled Jan 3 17:54:43 service103 kernel: igb 0000:08:00.1: eth1: PBA No: ffffff-0ff Jan 3 17:54:43 service103 ntpd[6989]: kernel time sync status 0040 Jan 3 17:54:43 service103 kernel: igb 0000:08:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Jan 3 17:54:43 service103 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Jan 3 17:54:43 service103 kernel: Type: CD-ROM ANSI SCSI revision: 03 Jan 3 17:54:43 service103 kernel: scsi 1:0:0:0: Attached scsi generic sg3 type 5 Jan 3 17:54:44 service103 kernel: sr0: scsi-1 drive Jan 3 17:54:44 service103 kernel: floppy0: no floppy controllers found Jan 3 17:54:44 service103 kernel: work still pending Jan 3 17:54:44 service103 kernel: lp: driver loaded but no devices found Jan 3 17:54:44 service103 kernel: ACPI: Power Button (FF) [PWRF] Jan 3 17:54:44 service103 kernel: ACPI: Power Button (CM) [PWRB] Jan 3 17:54:44 service103 kernel: ACPI: Mapper loaded Jan 3 17:54:44 service103 kernel: dell-wmi: No known WMI GUID found Jan 3 17:54:44 service103 kernel: md: Autodetecting RAID arrays. Jan 3 17:54:45 service103 kernel: md: autorun ... Jan 3 17:54:45 service103 kernel: md: ... autorun DONE. Jan 3 17:54:45 service103 kernel: device-mapper: multipath: version 1.0.6 loaded Jan 3 17:54:45 service103 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded Jan 3 17:54:45 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 17:54:45 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 17:54:45 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 17:54:45 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 17:54:45 service103 kernel: EXT3 FS on sda8, internal journal Jan 3 17:54:46 service103 kernel: kjournald starting. Commit interval 5 seconds Jan 3 17:54:46 service103 kernel: EXT3 FS on sda7, internal journal Jan 3 17:54:46 service103 kernel: EXT3-fs: mounted filesystem with ordered data mode. Jan 3 17:54:46 service103 kernel: Adding 2000052k swap on /dev/sda1. Priority:-1 extents:1 across:2000052k Jan 3 17:54:46 service103 leader-nodes-to-hosts-file: updating leader node entries in /etc/hosts... Jan 3 17:54:46 service103 kernel: IA-32 Microcode Update Driver: v1.14a Jan 3 17:54:46 service103 kernel: microcode: CPU1 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: microcode: CPU2 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: microcode: CPU4 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: microcode: CPU3 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: microcode: CPU7 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 ntpd[6989]: ntpd exiting on signal 15 Jan 3 17:54:47 service103 kernel: microcode: CPU6 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 ntpd[6989]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 17:54:47 service103 kernel: microcode: CPU5 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: microcode: CPU0 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 17:54:47 service103 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011) Jan 3 17:54:47 service103 kernel: NET: Registered protocol family 10 Jan 3 17:54:47 service103 kernel: lo: Disabled Privacy Extensions Jan 3 17:54:47 service103 kernel: IPv6 over IPv4 tunneling driver Jan 3 17:54:48 service103 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready Jan 3 17:54:48 service103 kernel: ib0: enabling connected mode will cause multicast packet drops Jan 3 17:54:48 service103 kernel: ib0: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib0: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ADDRCONF(NETDEV_UP): ib1: link is not ready Jan 3 17:54:48 service103 kernel: ib1: enabling connected mode will cause multicast packet drops Jan 3 17:54:48 service103 kernel: ib1: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib1: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib2: enabling connected mode will cause multicast packet drops Jan 3 17:54:48 service103 kernel: ib2: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib2: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib3: enabling connected mode will cause multicast packet drops Jan 3 17:54:48 service103 kernel: ib3: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: ib3: mtu > 2044 will cause multicast packet drops. Jan 3 17:54:48 service103 kernel: Loading iSCSI transport class v2.0-871. Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol cxgb3_register_client Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol t3_l2t_get Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol t3_l2e_free Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow Jan 3 17:54:49 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client Jan 3 17:54:49 service103 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client Jan 3 17:54:50 service103 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010) Jan 3 17:54:50 service103 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010) Jan 3 17:54:50 service103 kernel: iscsi: registered transport (bnx2i) Jan 3 17:54:50 service103 kernel: iscsi: registered transport (tcp) Jan 3 17:54:50 service103 kernel: iscsi: registered transport (be2iscsi) Jan 3 17:54:50 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 17:54:50 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 17:54:50 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 17:54:50 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 17:54:50 service103 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jan 3 17:54:50 service103 kernel: Bluetooth: Core ver 2.10 Jan 3 17:54:50 service103 kernel: NET: Registered protocol family 31 Jan 3 17:54:50 service103 kernel: Bluetooth: HCI device and connection manager initialized Jan 3 17:54:50 service103 kernel: Bluetooth: HCI socket layer initialized Jan 3 17:54:51 service103 kernel: Bluetooth: L2CAP ver 2.8 Jan 3 17:54:51 service103 kernel: Bluetooth: L2CAP socket layer initialized Jan 3 17:54:51 service103 kernel: Bluetooth: RFCOMM socket layer initialized Jan 3 17:54:51 service103 kernel: Bluetooth: RFCOMM TTY layer initialized Jan 3 17:54:51 service103 kernel: Bluetooth: RFCOMM ver 1.8 Jan 3 17:54:51 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jan 3 17:54:51 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_0 Jan 3 17:54:51 service103 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 Jan 3 17:54:51 service103 kernel: ipmi message handler version 39.1 Jan 3 17:54:51 service103 kernel: IPMI System Interface driver. Jan 3 17:54:51 service103 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 Jan 3 17:54:51 service103 kernel: ipmi: Found new BMC (man_id: 0x0028c5, prod_id: 0x0004, dev_id: 0x22) Jan 3 17:54:51 service103 kernel: IPMI kcs interface initialized Jan 3 17:54:52 service103 kernel: ipmi device interface Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Jan 3 17:54:52 service103 kernel: ADDRCONF(NETDEV_CHANGE): ib1: link becomes ready Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_0 Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_1 Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_1 Jan 3 17:54:52 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_1 Jan 3 17:54:52 service103 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Jan 3 17:54:52 service103 kernel: scsi2 : SRP.T10:1A6D0F0003C90200 Jan 3 17:54:52 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:52 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 17:54:53 service103 kernel: scsi 2:0:0:0: Attached scsi generic sg4 type 12 Jan 3 17:54:53 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:53 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:53 service103 kernel: sd 2:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 17:54:53 service103 kernel: sdb: Unit Not Ready, sense: Jan 3 17:54:53 service103 kernel: : Current: sense key: Unit Attention Jan 3 17:54:53 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 17:54:53 service103 kernel: Jan 3 17:54:53 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 17:54:54 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:54 service103 kernel: sdb: Write Protect is off Jan 3 17:54:54 service103 kernel: SCSI device sdb: drive cache: write back w/ FUA Jan 3 17:54:54 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 17:54:54 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:54 service103 kernel: sdb: Write Protect is off Jan 3 17:54:54 service103 kernel: SCSI device sdb: drive cache: write back w/ FUA Jan 3 17:54:54 service103 kernel: sdb: unknown partition table Jan 3 17:54:54 service103 kernel: sd 2:0:0:3: Attached scsi disk sdb Jan 3 17:54:54 service103 kernel: sd 2:0:0:3: Attached scsi generic sg5 type 0 Jan 3 17:54:54 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:54 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:54 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 17:54:54 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:55 service103 kernel: sdc: Write Protect is off Jan 3 17:54:55 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 17:54:55 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 17:54:55 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:55 service103 kernel: sdc: Write Protect is off Jan 3 17:54:55 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 17:54:55 service103 kernel: sdc: unknown partition table Jan 3 17:54:55 service103 kernel: sd 2:0:0:11: Attached scsi disk sdc Jan 3 17:54:55 service103 kernel: sd 2:0:0:11: Attached scsi generic sg6 type 0 Jan 3 17:54:55 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:56 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:56 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 17:54:56 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:56 service103 kernel: sdd: Write Protect is off Jan 3 17:54:56 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 17:54:56 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 17:54:56 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:56 service103 kernel: sdd: Write Protect is off Jan 3 17:54:57 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 17:54:57 service103 kernel: sdd: unknown partition table Jan 3 17:54:57 service103 kernel: sd 2:0:0:19: Attached scsi disk sdd Jan 3 17:54:57 service103 kernel: sd 2:0:0:19: Attached scsi generic sg7 type 0 Jan 3 17:54:57 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:57 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:57 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 17:54:57 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:57 service103 kernel: sde: Write Protect is off Jan 3 17:54:57 service103 kernel: SCSI device sde: drive cache: write back w/ FUA Jan 3 17:54:57 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 17:54:57 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:57 service103 kernel: sde: Write Protect is off Jan 3 17:54:57 service103 kernel: SCSI device sde: drive cache: write back w/ FUA Jan 3 17:54:57 service103 kernel: sde: unknown partition table Jan 3 17:54:58 service103 kernel: sd 2:0:0:27: Attached scsi disk sde Jan 3 17:54:58 service103 kernel: sd 2:0:0:27: Attached scsi generic sg8 type 0 Jan 3 17:54:58 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:58 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:58 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 17:54:58 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:58 service103 kernel: sdf: Write Protect is off Jan 3 17:54:58 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 17:54:58 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 17:54:58 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:58 service103 kernel: sdf: Write Protect is off Jan 3 17:54:58 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 17:54:59 service103 kernel: sdf: unknown partition table Jan 3 17:54:59 service103 kernel: sd 2:0:0:35: Attached scsi disk sdf Jan 3 17:54:59 service103 kernel: sd 2:0:0:35: Attached scsi generic sg9 type 0 Jan 3 17:54:59 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:54:59 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:54:59 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 17:54:59 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:59 service103 kernel: sdg: Write Protect is off Jan 3 17:54:59 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 17:54:59 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 17:54:59 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:54:59 service103 kernel: sdg: Write Protect is off Jan 3 17:55:00 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 17:55:00 service103 kernel: sdg: unknown partition table Jan 3 17:55:00 service103 kernel: sd 2:0:0:43: Attached scsi disk sdg Jan 3 17:55:00 service103 kernel: sd 2:0:0:43: Attached scsi generic sg10 type 0 Jan 3 17:55:00 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:00 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:00 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 17:55:00 service103 kernel: scsi3 : SRP.T10:56980F0003C90200 Jan 3 17:55:00 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:00 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 17:55:00 service103 kernel: scsi 3:0:0:0: Attached scsi generic sg11 type 12 Jan 3 17:55:00 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:00 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:01 service103 kernel: sd 3:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 17:55:01 service103 kernel: sdi: Unit Not Ready, sense: Jan 3 17:55:01 service103 kernel: : Current: sense key: Unit Attention Jan 3 17:55:01 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 17:55:01 service103 kernel: Jan 3 17:55:01 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 17:55:01 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:01 service103 kernel: sdi: Write Protect is off Jan 3 17:55:02 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 17:55:02 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 17:55:02 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:02 service103 kernel: sdi: Write Protect is off Jan 3 17:55:02 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 17:55:02 service103 kernel: sdi: unknown partition table Jan 3 17:55:02 service103 kernel: sd 3:0:0:3: Attached scsi disk sdi Jan 3 17:55:02 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:02 service103 kernel: sdh: Write Protect is off Jan 3 17:55:02 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 17:55:02 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 17:55:02 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:02 service103 kernel: sdh: Write Protect is off Jan 3 17:55:03 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 17:55:03 service103 kernel: sdh:<5>sd 3:0:0:3: Attached scsi generic sg12 type 0 Jan 3 17:55:03 service103 kernel: Vendor: SG unknown partition table Jan 3 17:55:03 service103 kernel: I<5>sd 2:0:0:51: Attached scsi disk sdh Jan 3 17:55:03 service103 kernel: sd 2:0:0:51: Attached scsi generic sg13 type 0 Jan 3 17:55:03 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:03 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:03 service103 kernel: sdj : very big device. try to use READ CAPACITY(16). Jan 3 17:55:03 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:03 service103 kernel: sdj: Write Protect is off Jan 3 17:55:03 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 17:55:03 service103 kernel: Model: <5>sdj : very big device. try to use READ CAPACITY(16). Jan 3 17:55:03 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:03 service103 kernel: sdj: Write Protect is off Jan 3 17:55:04 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 17:55:04 service103 kernel: sdj:DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:04 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:04 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 17:55:04 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:04 service103 kernel: unknown partition table Jan 3 17:55:04 service103 kernel: sd 2:0:0:59: Attached scsi disk sdj Jan 3 17:55:04 service103 kernel: sdk: Write Protect is off Jan 3 17:55:04 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 17:55:04 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 17:55:04 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:04 service103 kernel: sdk: Write Protect is off Jan 3 17:55:04 service103 kernel: sd 2:0:0:59: Attached scsi generic sg14 type 0 Jan 3 17:55:05 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:05 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:05 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 17:55:05 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:05 service103 kernel: sdl: Write Protect is off Jan 3 17:55:05 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 17:55:05 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 17:55:05 service103 kernel: sdk: unknown partition table Jan 3 17:55:05 service103 kernel: sd 3:0:0:11: Attached scsi disk sdk Jan 3 17:55:05 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 17:55:05 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:05 service103 kernel: sdl: Write Protect is off Jan 3 17:55:06 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 17:55:06 service103 kernel: sdl:<5>sd 3:0:0:11: Attached scsi generic sg15 type 0 Jan 3 17:55:06 service103 kernel: Vendor: SGI unknown partition table Jan 3 17:55:06 service103 kernel: <5>sd 2:0:0:67: Attached scsi disk sdl Jan 3 17:55:06 service103 kernel: sd 2:0:0:67: Attached scsi generic sg16 type 0 Jan 3 17:55:06 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:06 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:06 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 17:55:06 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:06 service103 kernel: sdm: Write Protect is off Jan 3 17:55:06 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 17:55:06 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 17:55:06 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:06 service103 kernel: sdm: Write Protect is off Jan 3 17:55:07 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 17:55:07 service103 kernel: sdm: Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:07 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 unknown partition table Jan 3 17:55:07 service103 kernel: Jan 3 17:55:07 service103 kernel: sd 2:0:0:75: Attached scsi disk sdm Jan 3 17:55:07 service103 kernel: sd 2:0:0:75: Attached scsi generic sg17 type 0 Jan 3 17:55:07 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:07 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:07 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 17:55:07 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:07 service103 kernel: sdn: Write Protect is off Jan 3 17:55:07 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 17:55:07 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 17:55:08 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:08 service103 kernel: sdn: Write Protect is off Jan 3 17:55:08 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 17:55:08 service103 kernel: sdn:<5>sdo : very big device. try to use READ CAPACITY(16). Jan 3 17:55:08 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:08 service103 kernel: unknown partition table Jan 3 17:55:08 service103 kernel: sd 2:0:0:83: Attached scsi disk sdn Jan 3 17:55:08 service103 kernel: sdo: Write Protect is off Jan 3 17:55:08 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 17:55:08 service103 kernel: sdo : very big device. try to use READ CAPACITY(16). Jan 3 17:55:08 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:08 service103 kernel: sdo: Write Protect is off Jan 3 17:55:09 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 17:55:09 service103 kernel: sdo: unknown partition table Jan 3 17:55:09 service103 kernel: sd 3:0:0:19: Attached scsi disk sdo Jan 3 17:55:09 service103 kernel: sd 2:0:0:83: Attached scsi generic sg18 type 0 Jan 3 17:55:09 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:09 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:09 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 17:55:09 service103 kernel: SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:09 service103 kernel: sdp: Write Protect is off Jan 3 17:55:10 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 17:55:10 service103 kernel: sd 3:0:0:19: Attached scsi generic sg19 type 0 Jan 3 17:55:10 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 17:55:10 service103 kernel: Vendor: S<5>SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:10 service103 kernel: G<5>sdp: Write Protect is off Jan 3 17:55:10 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 17:55:10 service103 kernel: sdp:I Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:10 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:10 service103 kernel: unknown partition table Jan 3 17:55:10 service103 ntpdate[7260]: step time server 172.29.0.1 offset -0.023036 sec Jan 3 17:55:10 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 17:55:10 service103 ntpd[7320]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) Jan 3 17:55:10 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:10 service103 ntpd[7321]: precision = 1.000 usec Jan 3 17:55:11 service103 kernel: sdq: Write Protect is off Jan 3 17:55:11 service103 ntpd[7321]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jan 3 17:55:11 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 17:55:11 service103 ntpd[7321]: Listening on interface wildcard, ::#123 Disabled Jan 3 17:55:11 service103 kernel: sd 2:0:0:91: Attached scsi disk sdp Jan 3 17:55:11 service103 ntpd[7321]: Listening on interface lo, ::1#123 Enabled Jan 3 17:55:11 service103 kernel: sd 2:0:0:91: Attached scsi generic sg20 type 0 Jan 3 17:55:12 service103 ntpd[7321]: Listening on interface ib1, fe80::202:c903:f:9f84#123 Enabled Jan 3 17:55:12 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 17:55:12 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:12 service103 kernel: sdq: Write Protect is off Jan 3 17:55:12 service103 ntpd[7321]: Listening on interface eth0, fe80::230:48ff:fec4:4f0c#123 Enabled Jan 3 17:55:12 service103 ntpd[7321]: Listening on interface ib0, fe80::202:c903:f:9f83#123 Enabled Jan 3 17:55:12 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 17:55:12 service103 ntpd[7321]: Listening on interface lo, 127.0.0.1#123 Enabled Jan 3 17:55:12 service103 kernel: sdq: unknown partition table Jan 3 17:55:12 service103 ntpd[7321]: Listening on interface eth0, 172.29.1.8#123 Enabled Jan 3 17:55:12 service103 kernel: sd 3:0:0:27: Attached scsi disk sdq Jan 3 17:55:13 service103 ntpd[7321]: Listening on interface ib0, 10.150.25.157#123 Enabled Jan 3 17:55:13 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:13 service103 ntpd[7321]: Listening on interface ib1, 10.151.25.157#123 Enabled Jan 3 17:55:13 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:13 service103 ntpd[7321]: kernel time sync status 0040 Jan 3 17:55:13 service103 kernel: sdr : very big device. try to use READ CAPACITY(16). Jan 3 17:55:13 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:13 service103 kernel: sdr: Write Protect is off Jan 3 17:55:13 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 17:55:13 service103 kernel: sd 3:0:0:27: Attached scsi generic sg21 type 0 Jan 3 17:55:13 service103 kernel: Vendor: SGI <5>sdr : very big device. try to use READ CAPACITY(16). Jan 3 17:55:13 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:13 service103 kernel: sdr: Write Protect is off Jan 3 17:55:14 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 17:55:14 service103 kernel: sdr: Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:14 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:14 service103 kernel: unknown partition table Jan 3 17:55:14 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 17:55:14 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:14 service103 kernel: sds: Write Protect is off Jan 3 17:55:14 service103 kernel: SCSI device sds: drive cache: write back w/ FUA Jan 3 17:55:14 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 17:55:14 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:14 service103 kernel: sds: Write Protect is off Jan 3 17:55:15 service103 kernel: sd 2:0:0:99: Attached scsi disk sdr Jan 3 17:55:15 service103 kernel: sd 2:0:0:99: Attached scsi generic sg22 type 0 Jan 3 17:55:15 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:15 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:15 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 17:55:15 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:15 service103 kernel: sdt: Write Protect is off Jan 3 17:55:15 service103 kernel: SCSI device sdt: drive cache: write back w/ FUA Jan 3 17:55:15 service103 kernel: SCSI device sds: drive cache: write back w/ FUA Jan 3 17:55:16 service103 kernel: sds: unknown partition table Jan 3 17:55:16 service103 kernel: sd 3:0:0:35: Attached scsi disk sds Jan 3 17:55:16 service103 kernel: sd 3:0:0:35: Attached scsi generic sg23 type 0 Jan 3 17:55:17 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 17:55:17 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:17 service103 kernel: sdt: Write Protect is off Jan 3 17:55:17 service103 kernel: SCSI device sdt: drive cache: write back w/ FUA Jan 3 17:55:17 service103 kernel: sdt:<5> Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:17 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 unknown partition table Jan 3 17:55:17 service103 kernel: Jan 3 17:55:17 service103 kernel: sd 2:0:0:107: Attached scsi disk sdt Jan 3 17:55:18 service103 kernel: sd 2:0:0:107: Attached scsi generic sg24 type 0 Jan 3 17:55:18 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:18 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:18 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 17:55:18 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:19 service103 kernel: sdu: Write Protect is off Jan 3 17:55:19 service103 kernel: SCSI device sdu: drive cache: write back w/ FUA Jan 3 17:55:19 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 17:55:19 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:20 service103 kernel: sdu: Write Protect is off Jan 3 17:55:20 service103 kernel: SCSI device sdu: drive cache: write back w/ FUA Jan 3 17:55:21 service103 kernel: sdu:<5>sdv : very big device. try to use READ CAPACITY(16). Jan 3 17:55:21 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:21 service103 kernel: unknown partition table Jan 3 17:55:21 service103 kernel: sd 2:0:0:115: Attached scsi disk sdu Jan 3 17:55:21 service103 kernel: sdv: Write Protect is off Jan 3 17:55:22 service103 kernel: SCSI device sdv: drive cache: write back w/ FUA Jan 3 17:55:22 service103 kernel: sd 2:0:0:115: Attached scsi generic sg25 type 0 Jan 3 17:55:22 service103 kernel: sdv : very big device. try to use READ CAPACITY(16). Jan 3 17:55:22 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:22 service103 kernel: sdv: Write Protect is off Jan 3 17:55:22 service103 kernel: SCSI device sdv: drive cache: write back w/ FUA Jan 3 17:55:22 service103 kernel: sdv: unknown partition table Jan 3 17:55:22 service103 kernel: sd 3:0:0:43: Attached scsi disk sdv Jan 3 17:55:22 service103 kernel: sd 3:0:0:43: Attached scsi generic sg26 type 0 Jan 3 17:55:23 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:23 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:23 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 17:55:23 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:23 service103 kernel: sdw: Write Protect is off Jan 3 17:55:23 service103 kernel: SCSI device sdw: drive cache: write back w/ FUA Jan 3 17:55:23 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 17:55:23 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:24 service103 kernel: sdw: Write Protect is off Jan 3 17:55:24 service103 kernel: SCSI device sdw: drive cache: write back w/ FUA Jan 3 17:55:24 service103 kernel: sdw: unknown partition table Jan 3 17:55:24 service103 kernel: sd 3:0:0:51: Attached scsi disk sdw Jan 3 17:55:24 service103 kernel: sd 3:0:0:51: Attached scsi generic sg27 type 0 Jan 3 17:55:24 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:24 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:24 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 17:55:24 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:24 service103 kernel: sdx: Write Protect is off Jan 3 17:55:25 service103 kernel: SCSI device sdx: drive cache: write back w/ FUA Jan 3 17:55:25 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 17:55:25 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:25 service103 kernel: sdx: Write Protect is off Jan 3 17:55:26 service103 kernel: SCSI device sdx: drive cache: write back w/ FUA Jan 3 17:55:26 service103 kernel: sdx: unknown partition table Jan 3 17:55:26 service103 kernel: sd 3:0:0:59: Attached scsi disk sdx Jan 3 17:55:26 service103 kernel: sd 3:0:0:59: Attached scsi generic sg28 type 0 Jan 3 17:55:27 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:27 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:27 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 17:55:27 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:27 service103 kernel: sdy: Write Protect is off Jan 3 17:55:27 service103 kernel: SCSI device sdy: drive cache: write back w/ FUA Jan 3 17:55:27 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 17:55:27 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:27 service103 kernel: sdy: Write Protect is off Jan 3 17:55:27 service103 kernel: SCSI device sdy: drive cache: write back w/ FUA Jan 3 17:55:27 service103 kernel: sdy: unknown partition table Jan 3 17:55:27 service103 kernel: sd 3:0:0:67: Attached scsi disk sdy Jan 3 17:55:27 service103 kernel: sd 3:0:0:67: Attached scsi generic sg29 type 0 Jan 3 17:55:27 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:27 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:28 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 17:55:28 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:28 service103 kernel: sdz: Write Protect is off Jan 3 17:55:28 service103 kernel: SCSI device sdz: drive cache: write back w/ FUA Jan 3 17:55:28 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 17:55:28 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:28 service103 kernel: sdz: Write Protect is off Jan 3 17:55:28 service103 kernel: SCSI device sdz: drive cache: write back w/ FUA Jan 3 17:55:28 service103 kernel: sdz: unknown partition table Jan 3 17:55:28 service103 kernel: sd 3:0:0:75: Attached scsi disk sdz Jan 3 17:55:28 service103 kernel: sd 3:0:0:75: Attached scsi generic sg30 type 0 Jan 3 17:55:28 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:28 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:29 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 17:55:29 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:29 service103 kernel: sdaa: Write Protect is off Jan 3 17:55:29 service103 kernel: SCSI device sdaa: drive cache: write back w/ FUA Jan 3 17:55:29 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 17:55:29 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:29 service103 kernel: sdaa: Write Protect is off Jan 3 17:55:29 service103 kernel: SCSI device sdaa: drive cache: write back w/ FUA Jan 3 17:55:29 service103 kernel: sdaa: unknown partition table Jan 3 17:55:29 service103 kernel: sd 3:0:0:83: Attached scsi disk sdaa Jan 3 17:55:29 service103 kernel: sd 3:0:0:83: Attached scsi generic sg31 type 0 Jan 3 17:55:29 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:30 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:30 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 17:55:30 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:30 service103 kernel: sdab: Write Protect is off Jan 3 17:55:30 service103 smartd[8448]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Jan 3 17:55:30 service103 kernel: SCSI device sdab: drive cache: write back w/ FUA Jan 3 17:55:30 service103 smartd[8448]: Home page is http://smartmontools.sourceforge.net/ Jan 3 17:55:30 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 17:55:30 service103 smartd[8448]: Opened configuration file /etc/smartd.conf Jan 3 17:55:30 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:30 service103 smartd[8448]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices Jan 3 17:55:30 service103 kernel: sdab: Write Protect is off Jan 3 17:55:30 service103 smartd[8448]: Device: /dev/hdb, opened Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/hdb, packet devices [this device CD/DVD] not SMART capable Jan 3 17:55:31 service103 kernel: SCSI device sdab: drive cache: write back w/ FUA Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sda, opened Jan 3 17:55:31 service103 kernel: sdab: unknown partition table Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sda, Bad IEC (SMART) mode page, err=4, skip device Jan 3 17:55:31 service103 kernel: sd 3:0:0:91: Attached scsi disk sdab Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdb, opened Jan 3 17:55:31 service103 kernel: sd 3:0:0:91: Attached scsi generic sg32 type 0 Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list. Jan 3 17:55:31 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdc, opened Jan 3 17:55:31 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list. Jan 3 17:55:31 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdd, opened Jan 3 17:55:31 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:31 service103 smartd[8448]: Device: /dev/sdd, is SMART capable. Adding to "monitor" list. Jan 3 17:55:32 service103 kernel: sdac: Write Protect is off Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sde, opened Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sde, is SMART capable. Adding to "monitor" list. Jan 3 17:55:32 service103 kernel: SCSI device sdac: drive cache: write back w/ FUA Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdf, opened Jan 3 17:55:32 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdf, is SMART capable. Adding to "monitor" list. Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdg, opened Jan 3 17:55:32 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdg, is SMART capable. Adding to "monitor" list. Jan 3 17:55:32 service103 kernel: sdac: Write Protect is off Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdh, opened Jan 3 17:55:32 service103 smartd[8448]: Device: /dev/sdh, is SMART capable. Adding to "monitor" list. Jan 3 17:55:33 service103 kernel: SCSI device sdac: drive cache: write back w/ FUA Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdi, opened Jan 3 17:55:33 service103 kernel: sdac: unknown partition table Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdi, is SMART capable. Adding to "monitor" list. Jan 3 17:55:33 service103 kernel: sd 3:0:0:99: Attached scsi disk sdac Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdj, opened Jan 3 17:55:33 service103 kernel: sd 3:0:0:99: Attached scsi generic sg33 type 0 Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdj, is SMART capable. Adding to "monitor" list. Jan 3 17:55:33 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdk, opened Jan 3 17:55:33 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdk, is SMART capable. Adding to "monitor" list. Jan 3 17:55:33 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdl, opened Jan 3 17:55:33 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:33 service103 smartd[8448]: Device: /dev/sdl, is SMART capable. Adding to "monitor" list. Jan 3 17:55:33 service103 kernel: sdad: Write Protect is off Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdm, opened Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdm, is SMART capable. Adding to "monitor" list. Jan 3 17:55:34 service103 kernel: SCSI device sdad: drive cache: write back w/ FUA Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdn, opened Jan 3 17:55:34 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdn, is SMART capable. Adding to "monitor" list. Jan 3 17:55:34 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdo, opened Jan 3 17:55:34 service103 kernel: sdad: Write Protect is off Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdo, is SMART capable. Adding to "monitor" list. Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdp, opened Jan 3 17:55:34 service103 kernel: SCSI device sdad: drive cache: write back w/ FUA Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdp, is SMART capable. Adding to "monitor" list. Jan 3 17:55:34 service103 kernel: sdad: unknown partition table Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdq, opened Jan 3 17:55:34 service103 kernel: sd 3:0:0:107: Attached scsi disk sdad Jan 3 17:55:34 service103 smartd[8448]: Device: /dev/sdq, is SMART capable. Adding to "monitor" list. Jan 3 17:55:35 service103 kernel: sd 3:0:0:107: Attached scsi generic sg34 type 0 Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdr, opened Jan 3 17:55:35 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdr, is SMART capable. Adding to "monitor" list. Jan 3 17:55:35 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sds, opened Jan 3 17:55:35 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sds, is SMART capable. Adding to "monitor" list. Jan 3 17:55:35 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdt, opened Jan 3 17:55:35 service103 kernel: sdae: Write Protect is off Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdt, is SMART capable. Adding to "monitor" list. Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdu, opened Jan 3 17:55:35 service103 kernel: SCSI device sdae: drive cache: write back w/ FUA Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdu, is SMART capable. Adding to "monitor" list. Jan 3 17:55:35 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 17:55:35 service103 smartd[8448]: Device: /dev/sdv, opened Jan 3 17:55:35 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdv, is SMART capable. Adding to "monitor" list. Jan 3 17:55:36 service103 kernel: sdae: Write Protect is off Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdw, opened Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdw, is SMART capable. Adding to "monitor" list. Jan 3 17:55:36 service103 kernel: SCSI device sdae: drive cache: write back w/ FUA Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdx, opened Jan 3 17:55:36 service103 kernel: sdae: unknown partition table Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdx, is SMART capable. Adding to "monitor" list. Jan 3 17:55:36 service103 kernel: sd 3:0:0:115: Attached scsi disk sdae Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdy, opened Jan 3 17:55:36 service103 kernel: sd 3:0:0:115: Attached scsi generic sg35 type 0 Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdy, is SMART capable. Adding to "monitor" list. Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdz, opened Jan 3 17:55:36 service103 smartd[8448]: Device: /dev/sdz, is SMART capable. Adding to "monitor" list. Jan 3 17:55:36 service103 smartd[8448]: Monitoring 0 ATA and 25 SCSI devices Jan 3 17:55:36 service103 smartd[8454]: smartd has fork()ed into background mode. New PID=8454. Jan 3 17:56:16 service103 ntpdate[8590]: step time server 172.29.0.1 offset -0.001857 sec Jan 3 17:56:16 service103 ntpd[8626]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) Jan 3 17:56:16 service103 ntpd[8627]: precision = 1.000 usec Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface wildcard, ::#123 Disabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface lo, ::1#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface ib1, fe80::202:c903:f:9f84#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface eth0, fe80::230:48ff:fec4:4f0c#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface ib0, fe80::202:c903:f:9f83#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface lo, 127.0.0.1#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface eth0, 172.29.1.8#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface ib0, 10.150.25.157#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: Listening on interface ib1, 10.151.25.157#123 Enabled Jan 3 17:56:16 service103 ntpd[8627]: kernel time sync status 0040 Jan 3 17:58:05 service103 kernel: init dynlocks cache Jan 3 17:58:05 service103 kernel: ldiskfs created from ext4-2.6-rhel5 Jan 3 17:58:09 service103 kernel: LDISKFS-fs (dm-1): recovery complete Jan 3 17:58:09 service103 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode Jan 3 17:58:09 service103 multipathd: dm-1: umount map (uevent) Jan 3 17:58:09 service103 kernel: JBD: barrier-based sync failed on dm-1-8 - disabling barriers Jan 3 17:58:10 service103 kernel: LDISKFS-fs (dm-13): recovery complete Jan 3 17:58:10 service103 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Jan 3 17:58:10 service103 multipathd: dm-13: umount map (uevent) Jan 3 17:58:10 service103 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Jan 3 17:58:11 service103 kernel: LDISKFS-fs (dm-14): recovery complete Jan 3 17:58:11 service103 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Jan 3 17:58:11 service103 multipathd: dm-14: umount map (uevent) Jan 3 17:58:11 service103 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Jan 3 17:58:13 service103 kernel: LDISKFS-fs (dm-2): recovery complete Jan 3 17:58:13 service103 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode Jan 3 17:58:13 service103 multipathd: dm-2: umount map (uevent) Jan 3 17:58:13 service103 kernel: JBD: barrier-based sync failed on dm-2-8 - disabling barriers Jan 3 17:58:17 service103 kernel: LDISKFS-fs (dm-0): recovery complete Jan 3 17:58:17 service103 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode Jan 3 17:58:17 service103 multipathd: dm-0: umount map (uevent) Jan 3 17:58:17 service103 kernel: JBD: barrier-based sync failed on dm-0-8 - disabling barriers Jan 3 17:58:19 service103 kernel: LDISKFS-fs (dm-3): recovery complete Jan 3 17:58:19 service103 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode Jan 3 17:58:19 service103 multipathd: dm-3: umount map (uevent) Jan 3 17:58:19 service103 kernel: JBD: barrier-based sync failed on dm-3-8 - disabling barriers Jan 3 17:58:20 service103 kernel: LDISKFS-fs (dm-4): recovery complete Jan 3 17:58:20 service103 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode Jan 3 17:58:20 service103 multipathd: dm-4: umount map (uevent) Jan 3 17:58:20 service103 kernel: JBD: barrier-based sync failed on dm-4-8 - disabling barriers Jan 3 17:58:22 service103 kernel: LDISKFS-fs (dm-5): recovery complete Jan 3 17:58:22 service103 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Jan 3 17:58:22 service103 multipathd: dm-5: umount map (uevent) Jan 3 17:58:22 service103 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Jan 3 17:58:25 service103 kernel: LDISKFS-fs (dm-6): recovery complete Jan 3 17:58:25 service103 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Jan 3 17:58:25 service103 multipathd: dm-6: umount map (uevent) Jan 3 17:58:25 service103 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Jan 3 17:58:28 service103 kernel: LDISKFS-fs (dm-7): recovery complete Jan 3 17:58:28 service103 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode Jan 3 17:58:28 service103 multipathd: dm-7: umount map (uevent) Jan 3 17:58:28 service103 kernel: JBD: barrier-based sync failed on dm-7-8 - disabling barriers Jan 3 17:58:32 service103 kernel: LDISKFS-fs (dm-8): recovery complete Jan 3 17:58:32 service103 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode Jan 3 17:58:32 service103 multipathd: dm-8: umount map (uevent) Jan 3 17:58:32 service103 kernel: JBD: barrier-based sync failed on dm-8-8 - disabling barriers Jan 3 17:58:35 service103 kernel: LDISKFS-fs (dm-9): recovery complete Jan 3 17:58:35 service103 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Jan 3 17:58:35 service103 multipathd: dm-9: umount map (uevent) Jan 3 17:58:35 service103 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Jan 3 17:58:39 service103 kernel: LDISKFS-fs (dm-10): recovery complete Jan 3 17:58:39 service103 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Jan 3 17:58:39 service103 multipathd: dm-10: umount map (uevent) Jan 3 17:58:39 service103 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Jan 3 17:58:42 service103 kernel: LDISKFS-fs (dm-11): recovery complete Jan 3 17:58:42 service103 kernel: LDISKFS-fs (dm-11): mounted filesystem with ordered data mode Jan 3 17:58:42 service103 multipathd: dm-11: umount map (uevent) Jan 3 17:58:42 service103 kernel: JBD: barrier-based sync failed on dm-11-8 - disabling barriers Jan 3 17:58:43 service103 kernel: LDISKFS-fs (dm-12): recovery complete Jan 3 17:58:43 service103 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode Jan 3 17:58:43 service103 multipathd: dm-12: umount map (uevent) Jan 3 17:58:43 service103 kernel: JBD: barrier-based sync failed on dm-12-8 - disabling barriers Jan 3 17:59:13 service103 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Jan 3 17:59:14 service103 multipathd: dm-8: umount map (uevent) Jan 3 17:59:14 service103 kernel: Lustre: Lustre Version: 1.8.6.81 Jan 3 17:59:15 service103 kernel: Lustre: Build Version: lustre/scripts-1.8.6 Jan 3 17:59:16 service103 kernel: Lustre: Listener bound to ib1:10.151.25.157:987:mlx4_0 Jan 3 17:59:16 service103 multipathd: dm-13: umount map (uevent) Jan 3 17:59:16 service103 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Jan 3 17:59:16 service103 multipathd: dm-10: umount map (uevent) Jan 3 17:59:16 service103 kernel: Lustre: Added LNI 10.151.25.157@o2ib [8/64/0/180] Jan 3 17:59:16 service103 multipathd: dm-3: umount map (uevent) Jan 3 17:59:16 service103 kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Jan 3 17:59:17 service103 multipathd: dm-9: umount map (uevent) Jan 3 17:59:17 service103 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Jan 3 17:59:17 service103 multipathd: dm-14: umount map (uevent) Jan 3 17:59:17 service103 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Jan 3 17:59:17 service103 multipathd: dm-2: umount map (uevent) Jan 3 17:59:17 service103 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode Jan 3 17:59:17 service103 multipathd: dm-4: umount map (uevent) Jan 3 17:59:17 service103 kernel: JBD: barrier-based sync failed on dm-8-8 - disabling barriers Jan 3 17:59:17 service103 multipathd: dm-6: umount map (uevent) Jan 3 17:59:17 service103 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Jan 3 17:59:17 service103 multipathd: dm-11: umount map (uevent) Jan 3 17:59:17 service103 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Jan 3 17:59:17 service103 multipathd: dm-12: umount map (uevent) Jan 3 17:59:17 service103 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode Jan 3 17:59:17 service103 multipathd: dm-5: umount map (uevent) Jan 3 17:59:18 service103 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Jan 3 17:59:18 service103 multipathd: dm-7: umount map (uevent) Jan 3 17:59:18 service103 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Jan 3 17:59:18 service103 multipathd: dm-1: umount map (uevent) Jan 3 17:59:18 service103 kernel: Lustre: MGC10.151.25.163@o2ib: Reactivating import Jan 3 17:59:18 service103 multipathd: dm-0: umount map (uevent) Jan 3 17:59:18 service103 kernel: JBD: barrier-based sync failed on dm-8-8 - disabling barriers Jan 3 17:59:18 service103 kernel: Lustre: Filtering OBD driver; http://wiki.whamcloud.com/ Jan 3 17:59:18 service103 kernel: Lustre: 10331:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST0042, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776061346 Jan 3 17:59:19 service103 kernel: Lustre: nbp6-OST0042: Now serving nbp6-OST0042 on /dev/mapper/ddn6a-nbp6-ost66 with recovery enabled Jan 3 17:59:19 service103 kernel: Lustre: nbp6-OST0042: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:19 service103 kernel: LustreError: 10331:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST0042: unknown param writethrough=0 Jan 3 17:59:19 service103 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Jan 3 17:59:19 service103 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode Jan 3 17:59:19 service103 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Jan 3 17:59:19 service103 kernel: JBD: barrier-based sync failed on dm-3-8 - disabling barriers Jan 3 17:59:19 service103 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Jan 3 17:59:19 service103 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Jan 3 17:59:20 service103 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: JBD: barrier-based sync failed on dm-2-8 - disabling barriers Jan 3 17:59:20 service103 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Jan 3 17:59:20 service103 kernel: JBD: barrier-based sync failed on dm-4-8 - disabling barriers Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-11): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: JBD: barrier-based sync failed on dm-11-8 - disabling barriers Jan 3 17:59:21 service103 kernel: JBD: barrier-based sync failed on dm-12-8 - disabling barriers Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode Jan 3 17:59:21 service103 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-11): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: JBD: barrier-based sync failed on dm-7-8 - disabling barriers Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: JBD: barrier-based sync failed on dm-1-8 - disabling barriers Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: JBD: barrier-based sync failed on dm-0-8 - disabling barriers Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode Jan 3 17:59:22 service103 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode Jan 3 17:59:23 service103 kernel: Lustre: 10426:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST0052, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776184966 Jan 3 17:59:23 service103 kernel: Lustre: nbp6-OST0052: Now serving nbp6-OST0052 on /dev/mapper/ddn6a-nbp6-ost82 with recovery enabled Jan 3 17:59:23 service103 kernel: Lustre: nbp6-OST0052: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:23 service103 kernel: LustreError: 10426:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST0052: unknown param writethrough=0 Jan 3 17:59:23 service103 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Jan 3 17:59:23 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST0002_UUID' is not available for connect (no target) Jan 3 17:59:23 service103 kernel: LustreError: 10051:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c056b0400 x1389011668723017/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1325642457 ref 1 fl Interpret:/0/0 rc -19/0 Jan 3 17:59:23 service103 kernel: Lustre: 10914:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST006a, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776067253 Jan 3 17:59:24 service103 kernel: Lustre: nbp6-OST006a: Now serving nbp6-OST006a on /dev/mapper/ddn6a-nbp6-ost106 with recovery enabled Jan 3 17:59:24 service103 kernel: Lustre: nbp6-OST006a: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:24 service103 kernel: LustreError: 10914:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST006a: unknown param writethrough=0 Jan 3 17:59:24 service103 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Jan 3 17:59:24 service103 kernel: JBD: barrier-based sync failed on dm-3-8 - disabling barriers Jan 3 17:59:24 service103 kernel: Lustre: 10972:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST001a, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776335828 Jan 3 17:59:24 service103 kernel: Lustre: 10972:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Jan 3 17:59:24 service103 kernel: Lustre: nbp6-OST001a: Now serving nbp6-OST001a on /dev/mapper/ddn6a-nbp6-ost26 with recovery enabled Jan 3 17:59:24 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 17:59:24 service103 kernel: Lustre: nbp6-OST001a: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:25 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 17:59:25 service103 kernel: LustreError: 10972:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST001a: unknown param writethrough=0 Jan 3 17:59:25 service103 kernel: LustreError: 10972:0:(obd_config.c:1011:class_process_proc_param()) Skipped 1 previous similar message Jan 3 17:59:25 service103 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Jan 3 17:59:25 service103 kernel: JBD: barrier-based sync failed on dm-2-8 - disabling barriers Jan 3 17:59:25 service103 kernel: JBD: barrier-based sync failed on dm-4-8 - disabling barriers Jan 3 17:59:25 service103 kernel: Lustre: 10989:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST0022, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776147314 Jan 3 17:59:25 service103 kernel: Lustre: 10989:0:(filter.c:1001:filter_init_server_data()) Skipped 2 previous similar messages Jan 3 17:59:25 service103 kernel: Lustre: nbp6-OST0022: Now serving nbp6-OST0022 on /dev/mapper/ddn6a-nbp6-ost34 with recovery enabled Jan 3 17:59:25 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 17:59:25 service103 kernel: Lustre: nbp6-OST0022: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:25 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 17:59:26 service103 kernel: LustreError: 10989:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST0022: unknown param writethrough=0 Jan 3 17:59:26 service103 kernel: LustreError: 10989:0:(obd_config.c:1011:class_process_proc_param()) Skipped 2 previous similar messages Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-11-8 - disabling barriers Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-12-8 - disabling barriers Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-7-8 - disabling barriers Jan 3 17:59:26 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST000a_UUID' is not available for connect (no target) Jan 3 17:59:26 service103 kernel: Lustre: 10062:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0012: 11919 recoverable clients remain Jan 3 17:59:26 service103 kernel: LustreError: 10052:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c01f88400 x1389485741634836/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1325642466 ref 1 fl Interpret:/0/0 rc -19/0 Jan 3 17:59:26 service103 kernel: JBD: barrier-based sync failed on dm-1-8 - disabling barriers Jan 3 17:59:27 service103 kernel: Lustre: 11071:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service nbp6-OST000a, 11920 recoverable clients, 0 delayed clients, last_rcvd 25776129373 Jan 3 17:59:27 service103 kernel: Lustre: 11071:0:(filter.c:1001:filter_init_server_data()) Skipped 5 previous similar messages Jan 3 17:59:27 service103 kernel: Lustre: nbp6-OST000a: Now serving nbp6-OST000a on /dev/mapper/ddn6a-nbp6-ost10 with recovery enabled Jan 3 17:59:27 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 17:59:27 service103 kernel: Lustre: nbp6-OST000a: Will be in recovery for at least 5:00, or until 11920 clients reconnect Jan 3 17:59:27 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 17:59:27 service103 kernel: LustreError: 11071:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST000a: unknown param writethrough=0 Jan 3 17:59:27 service103 kernel: LustreError: 11071:0:(obd_config.c:1011:class_process_proc_param()) Skipped 5 previous similar messages Jan 3 17:59:27 service103 kernel: JBD: barrier-based sync failed on dm-0-8 - disabling barriers Jan 3 17:59:28 service103 kernel: Lustre: 10058:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST000a: 11919 recoverable clients remain Jan 3 17:59:28 service103 kernel: Lustre: 10058:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 12 previous similar messages Jan 3 17:59:29 service103 kernel: Lustre: 10064:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0042: 11918 recoverable clients remain Jan 3 17:59:29 service103 kernel: Lustre: 10064:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 11 previous similar messages Jan 3 17:59:32 service103 ntpd[8627]: synchronized to 172.29.0.1, stratum 3 Jan 3 17:59:42 service103 kernel: Lustre: 10072:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0062: 11899 recoverable clients remain Jan 3 17:59:49 service103 kernel: Lustre: 10076:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0002: 11900 recoverable clients remain Jan 3 17:59:56 service103 kernel: Lustre: 10082:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0002: 11899 recoverable clients remain Jan 3 18:00:20 service103 kernel: Lustre: 10098:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0002: 11897 recoverable clients remain Jan 3 18:00:20 service103 kernel: Lustre: 10098:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 2 previous similar messages Jan 3 18:00:36 service103 kernel: Lustre: 10107:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0062: 11897 recoverable clients remain Jan 3 18:00:36 service103 kernel: Lustre: 10107:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 9 previous similar messages Jan 3 18:00:45 service103 kernel: LustreError: 10125:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.51.3@o2ib (b8520454-5861-29ba-cd63-dfaa9277de5f): 11896 clients in recovery for 761s Jan 3 18:00:45 service103 kernel: LustreError: 10121:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c12becc50 x1390027882005719/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642545 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:00:45 service103 kernel: LustreError: 10139:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.51.2@o2ib (c61ee50b-3655-0e28-5cf5-0385ac735bd0): 11896 clients in recovery for 761s Jan 3 18:00:45 service103 kernel: LustreError: 10139:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 1 previous similar message Jan 3 18:00:45 service103 kernel: LustreError: 10125:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 1 previous similar message Jan 3 18:00:48 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.44.32@o2ib (6c6fc931-ec9b-e6fd-d32f-a1c928bba5bd): 11890 clients in recovery for 758s Jan 3 18:00:48 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c12bf4c50 x1390027880956847/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642548 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:00:48 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jan 3 18:00:49 service103 kernel: LustreError: 10072:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.44.53@o2ib (de10a58c-e643-339a-7b6d-cc156c6a1da0): 11877 clients in recovery for 757s Jan 3 18:00:49 service103 kernel: LustreError: 10072:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 6 previous similar messages Jan 3 18:00:50 service103 kernel: LustreError: 10118:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b3a868000 x1390027880956861/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642550 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:00:50 service103 kernel: LustreError: 10118:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18 previous similar messages Jan 3 18:00:52 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0062: denying connection for new client 10.151.52.145@o2ib (adc707f1-ddc9-3b13-a76d-4ecbcd5ca8a6): 11841 clients in recovery for 730s Jan 3 18:00:52 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 12 previous similar messages Jan 3 18:01:08 service103 kernel: Lustre: 10170:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0062: 11672 recoverable clients remain Jan 3 18:01:08 service103 kernel: Lustre: 10170:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 3631 previous similar messages Jan 3 18:01:09 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.50.179@o2ib (bcb4dea8-923d-76f5-6860-589cd1a176b3): 11799 clients in recovery for 737s Jan 3 18:01:09 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810abd37a850 x1390027879929815/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642569 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:01:09 service103 kernel: LustreError: 10131:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jan 3 18:01:39 service103 kernel: LustreError: 10874:0:(filter_log.c:135:filter_cancel_cookies_cb()) error cancelling log cookies: rc = -19 Jan 3 18:01:39 service103 kernel: LustreError: 10874:0:(filter_log.c:135:filter_cancel_cookies_cb()) error cancelling log cookies: rc = -19 Jan 3 18:02:12 service103 kernel: Lustre: 10138:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0002: 11757 recoverable clients remain Jan 3 18:02:12 service103 kernel: Lustre: 10138:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 962 previous similar messages Jan 3 18:03:02 service103 kernel: LustreError: 10091:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.32.67@o2ib (12fb0d8a-0624-9476-0f6f-9b0449cb9cf1): 11738 clients in recovery for 624s Jan 3 18:03:02 service103 kernel: LustreError: 10071:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8109f55d3800 x1390035154798112/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642682 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:03:02 service103 kernel: LustreError: 10071:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jan 3 18:03:02 service103 kernel: LustreError: 10091:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 17 previous similar messages Jan 3 18:03:21 service103 kernel: LustreError: 10134:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.51.3@o2ib (b8520454-5861-29ba-cd63-dfaa9277de5f): 11576 clients in recovery for 604s Jan 3 18:03:21 service103 kernel: LustreError: 10113:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8109f1557800 x1390027882009054/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642701 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:03:21 service103 kernel: LustreError: 10113:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 13 previous similar messages Jan 3 18:03:21 service103 kernel: LustreError: 10134:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 3 previous similar messages Jan 3 18:04:09 service103 kernel: LustreError: 10122:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST000a: denying connection for new client 10.151.6.156@o2ib (75b74635-7b6f-146d-5623-2841ff309de3): 34 clients in recovery for 536s Jan 3 18:04:09 service103 kernel: LustreError: 10151:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810920a67400 x1390035605678822/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642748 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:04:09 service103 kernel: LustreError: 10151:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38 previous similar messages Jan 3 18:04:09 service103 kernel: LustreError: 10122:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 48 previous similar messages Jan 3 18:04:22 service103 kernel: Lustre: 10110:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) nbp6-OST0002: 29 recoverable clients remain Jan 3 18:04:22 service103 kernel: Lustre: 10110:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 173111 previous similar messages Jan 3 18:05:57 service103 kernel: LustreError: 10144:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.32.67@o2ib (12fb0d8a-0624-9476-0f6f-9b0449cb9cf1): 29 clients in recovery for 449s Jan 3 18:05:57 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST000a: denying connection for new client 10.151.32.67@o2ib (12fb0d8a-0624-9476-0f6f-9b0449cb9cf1): 29 clients in recovery for 428s Jan 3 18:05:57 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 105 previous similar messages Jan 3 18:05:57 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81092005d400 x1390035154801445/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325642857 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:05:57 service103 kernel: LustreError: 10105:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 117 previous similar messages Jan 3 18:05:57 service103 kernel: LustreError: 10144:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 13 previous similar messages Jan 3 18:08:34 service103 kernel: LustreError: 10102:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0002: denying connection for new client 10.151.51.3@o2ib (b8520454-5861-29ba-cd63-dfaa9277de5f): 29 clients in recovery for 292s Jan 3 18:08:34 service103 kernel: LustreError: 10136:0:(ldlm_lib.c:944:target_handle_connect()) nbp6-OST0062: denying connection for new client 10.151.51.3@o2ib (b8520454-5861-29ba-cd63-dfaa9277de5f): 29 clients in recovery for 269s Jan 3 18:08:34 service103 kernel: LustreError: 10136:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 156 previous similar messages Jan 3 18:08:34 service103 kernel: LustreError: 10136:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810922374800 x1390027882014746/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325643013 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:08:34 service103 kernel: LustreError: 10136:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 170 previous similar messages Jan 3 18:08:34 service103 kernel: LustreError: 10102:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 2 previous similar messages Jan 3 18:12:54 service103 kernel: LustreError: 10138:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810920d78800 x1390035605688829/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325643273 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:12:54 service103 kernel: LustreError: 10128:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810920281c00 x1390035605688837/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325643273 ref 1 fl Interpret:/0/0 rc -16/0 Jan 3 18:12:54 service103 kernel: LustreError: 10128:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 225 previous similar messages Jan 3 18:12:54 service103 kernel: LustreError: 10138:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10 previous similar messages Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0012: Recovery period over after 13:37, of 11920 clients 11891 recovered and 29 were evicted. Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0012: sending delayed replies to recovered clients Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0012: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:03 service103 kernel: Lustre: 10071:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST0012: deleting orphan objects from 278729 to 278753, orphan objids won't be reused any more. Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0062: Recovery period over after 13:37, of 11901 clients 11872 recovered and 29 were evicted. Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0062: sending delayed replies to recovered clients Jan 3 18:13:03 service103 kernel: LustreError: 10785:0:(filter_log.c:135:filter_cancel_cookies_cb()) error cancelling log cookies: rc = -19 Jan 3 18:13:03 service103 kernel: LustreError: 10785:0:(filter_log.c:135:filter_cancel_cookies_cb()) Skipped 1 previous similar message Jan 3 18:13:03 service103 kernel: Lustre: nbp6-OST0062: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:03 service103 kernel: Lustre: 10171:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST0062: deleting orphan objects from 278712 to 278721, orphan objids won't be reused any more. Jan 3 18:13:04 service103 kernel: Lustre: nbp6-OST004a: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:04 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 18:13:04 service103 kernel: Lustre: 10065:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST004a: deleting orphan objects from 278726 to 278753, orphan objids won't be reused any more. Jan 3 18:13:04 service103 kernel: Lustre: 10065:0:(filter.c:3126:filter_destroy_precreated()) Skipped 1 previous similar message Jan 3 18:13:04 service103 kernel: Lustre: nbp6-OST0042: Recovery period over after 13:38, of 11920 clients 11891 recovered and 29 were evicted. Jan 3 18:13:04 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 18:13:04 service103 kernel: Lustre: nbp6-OST0042: sending delayed replies to recovered clients Jan 3 18:13:04 service103 kernel: Lustre: Skipped 2 previous similar messages Jan 3 18:13:04 service103 kernel: LustreError: 9985:0:(filter_log.c:135:filter_cancel_cookies_cb()) error cancelling log cookies: rc = -19 Jan 3 18:13:05 service103 kernel: Lustre: nbp6-OST006a: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:05 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:13:05 service103 kernel: Lustre: 10103:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST006a: deleting orphan objects from 278667 to 278689, orphan objids won't be reused any more. Jan 3 18:13:05 service103 kernel: Lustre: nbp6-OST000a: Recovery period over after 13:37, of 11920 clients 11891 recovered and 29 were evicted. Jan 3 18:13:05 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:13:05 service103 kernel: Lustre: nbp6-OST000a: sending delayed replies to recovered clients Jan 3 18:13:05 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:13:05 service103 kernel: Lustre: 10103:0:(filter.c:3126:filter_destroy_precreated()) Skipped 3 previous similar messages Jan 3 18:13:06 service103 kernel: LustreError: 10688:0:(filter_log.c:135:filter_cancel_cookies_cb()) error cancelling log cookies: rc = -19 Jan 3 18:13:06 service103 kernel: LustreError: 10688:0:(filter_log.c:135:filter_cancel_cookies_cb()) Skipped 2 previous similar messages Jan 3 18:13:07 service103 kernel: Lustre: nbp6-OST001a: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:07 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 18:13:07 service103 kernel: Lustre: 10121:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST001a: deleting orphan objects from 278726 to 278753, orphan objids won't be reused any more. Jan 3 18:13:07 service103 kernel: Lustre: 10121:0:(filter.c:3126:filter_destroy_precreated()) Skipped 5 previous similar messages Jan 3 18:13:19 service103 pcp-pmie[8185]: High 1-minute load average 40.5load@service103 Jan 3 18:13:26 service103 kernel: Lustre: nbp6-OST0002: Recovery period over after 13:37, of 11901 clients 11872 recovered and 29 were evicted. Jan 3 18:13:26 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 18:13:26 service103 kernel: Lustre: nbp6-OST0002: sending delayed replies to recovered clients Jan 3 18:13:26 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 18:13:27 service103 kernel: Lustre: nbp6-OST0002: received MDS connection from 10.151.25.163@o2ib Jan 3 18:13:27 service103 kernel: Lustre: 10065:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST0002: deleting orphan objects from 278910 to 278913, orphan objids won't be reused any more. Jan 3 18:15:33 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 10.151.32.139@o2ib ns: filter-nbp6-OST0012_UUID lock: ffff8109cb3ece00/0x69beea547f7c3e46 lrc: 3/0,0 mode: PW/PW res: 277980/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x39a004e713d5efa2 expref: 4 pid: 10089 timeout 4296043457 Jan 3 18:15:33 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 10.151.59.80@o2ib ns: filter-nbp6-OST0062_UUID lock: ffff810b70d91a00/0x69beea547f78869a lrc: 3/0,0 mode: PW/PW res: 278710/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xf597e8feece4b8b0 expref: 29 pid: 10055 timeout 4296043720 Jan 3 18:15:35 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 10.151.22.202@o2ib ns: filter-nbp6-OST000a_UUID lock: ffff8109b92c5c00/0x69beea547f7e0029 lrc: 3/0,0 mode: PR/PR res: 258980/0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xdd264cadd57a2ab8 expref: 4 pid: 10113 timeout 4296045998 Jan 3 18:15:37 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 10.151.38.172@o2ib ns: filter-nbp6-OST0022_UUID lock: ffff8109be0aca00/0x69beea547f7d59b8 lrc: 3/0,0 mode: PW/PW res: 278657/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x1337ff487d2d48fc expref: 9 pid: 10143 timeout 4296047499 Jan 3 18:15:37 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) Skipped 3 previous similar messages Jan 3 18:15:37 service103 kernel: LustreError: 18784:0:(filter.c:1557:filter_destroy_internal()) destroying objid 278683 ino 4430521 nlink 0 count 2 Jan 3 18:15:37 service103 kernel: LustreError: 18779:0:(filter.c:1557:filter_destroy_internal()) destroying objid 278657 ino 6491468 nlink 0 count 2 Jan 3 18:15:37 service103 kernel: LustreError: 18779:0:(filter.c:1563:filter_destroy_internal()) error unlinking objid 278657: rc -2 Jan 3 18:15:38 service103 kernel: LustreError: 18784:0:(filter.c:1563:filter_destroy_internal()) error unlinking objid 278683: rc -2 Jan 3 18:15:57 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 10.151.59.144@o2ib ns: filter-nbp6-OST0002_UUID lock: ffff8109f5d1a200/0x69beea547f788d54 lrc: 3/0,0 mode: PW/PW res: 278902/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xc973ed121cf6eadf expref: 7 pid: 10116 timeout 4296067371 Jan 3 18:15:57 service103 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) Skipped 1 previous similar message Jan 3 18:16:08 service103 kernel: Lustre: 19535:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff810ac8563800 already connecting Jan 3 18:16:08 service103 kernel: Lustre: 19535:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 3 previous similar messages Jan 3 18:16:10 service103 kernel: Lustre: 19550:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff81093bddfc00 already connecting Jan 3 18:16:10 service103 kernel: Lustre: 19550:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 3 previous similar messages Jan 3 18:16:11 service103 kernel: Lustre: 19321:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff810949b54a00 already connecting Jan 3 18:16:11 service103 kernel: Lustre: 19321:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 3 previous similar messages Jan 3 18:16:13 service103 kernel: Lustre: 19550:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff810b94e79800 already connecting Jan 3 18:16:13 service103 kernel: Lustre: 19550:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 7 previous similar messages Jan 3 18:16:23 service103 kernel: Lustre: Service thread pid 10186 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:16:23 service103 kernel: Pid: 10186, comm: ll_ost_io_05 Jan 3 18:16:23 service103 kernel: Jan 3 18:16:23 service103 kernel: Call Trace: Jan 3 18:16:23 service103 kernel: [] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jan 3 18:16:23 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 18:16:23 service103 kernel: [] __down_write+0xb/0xd Jan 3 18:16:23 service103 kernel: [] down_write+0x11/0x13 Jan 3 18:16:23 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 18:16:23 service103 kernel: [] filter_destroy+0x99d/0x1fb0 [obdfilter] Jan 3 18:16:23 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 18:16:23 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 18:16:23 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 18:16:26 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 18:16:26 service103 kernel: [] __ldlm_handle2lock+0x8c/0x360 [ptlrpc] Jan 3 18:16:26 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 18:16:26 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 18:16:26 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 18:16:26 service103 kernel: [] list_add+0xc/0xe Jan 3 18:16:26 service103 kernel: [] __rmqueue+0x91/0xc6 Jan 3 18:16:26 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:16:26 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:16:27 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:16:27 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:16:27 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:16:27 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:16:27 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:16:27 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:16:27 service103 kernel: Jan 3 18:16:27 service103 kernel: Lustre: Service thread pid 10225 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:16:27 service103 kernel: Pid: 10225, comm: ll_ost_io_44 Jan 3 18:16:27 service103 kernel: Jan 3 18:16:28 service103 kernel: Call Trace: Jan 3 18:16:28 service103 kernel: [] filter_do_bio+0xaf2/0xbb0 [obdfilter] Jan 3 18:16:28 service103 kernel: [] __ldiskfs_journal_stop+0x78/0xa0 [ldiskfs] Jan 3 18:16:28 service103 kernel: [] fsfilt_ldiskfs_write_record+0x412/0x460 [fsfilt_ldiskfs] Jan 3 18:16:28 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:16:28 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:16:28 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:16:28 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 18:16:28 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:16:29 service103 kernel: [] filter_quota_getflag+0x655/0x8bb [lquota] Jan 3 18:16:29 service103 kernel: [] filter_commitrw_write+0x1b77/0x2dc0 [obdfilter] Jan 3 18:16:29 service103 kernel: [] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jan 3 18:16:29 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:16:29 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 18:16:29 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 18:16:29 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 18:16:30 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 18:16:30 service103 kernel: [] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] ldlm_resource_putref+0xb/0x10 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] ldlm_resource_iterate+0x1ad/0x240 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:16:30 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:16:30 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:16:31 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:16:31 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:16:31 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:16:31 service103 kernel: Jan 3 18:16:31 service103 kernel: Pid: 10115, comm: ll_ost_64 Jan 3 18:16:31 service103 kernel: Jan 3 18:16:31 service103 kernel: Call Trace: Jan 3 18:16:31 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:16:31 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:16:31 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:16:32 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:16:32 service103 kernel: [] ldiskfs_acquire_dquot+0x64/0xb0 [ldiskfs] Jan 3 18:16:32 service103 kernel: [] dqget+0x286/0x2b6 Jan 3 18:16:32 service103 kernel: [] vfs_get_dqblk+0x31/0xcf Jan 3 18:16:32 service103 kernel: [] fsfilt_ldiskfs_quotactl+0x96e/0xf60 [fsfilt_ldiskfs] Jan 3 18:16:32 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:16:32 service103 kernel: [] filter_quota_ctl+0x272/0xbf0 [lquota] Jan 3 18:16:32 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 18:16:32 service103 kernel: [] lustre_swab_buf+0x81/0x170 [ptlrpc] Jan 3 18:16:32 service103 kernel: [] ost_handle+0x478b/0x55b0 [ost] Jan 3 18:16:33 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:16:33 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:16:33 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:16:33 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:16:33 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:16:33 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:16:33 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:16:33 service103 kernel: Jan 3 18:16:33 service103 kernel: Lustre: Service thread pid 10185 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:16:33 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 18:16:33 service103 kernel: Pid: 10185, comm: ll_ost_io_04 Jan 3 18:16:34 service103 kernel: Jan 3 18:16:34 service103 kernel: Call Trace: Jan 3 18:16:34 service103 kernel: [] ldiskfs_mb_free_blocks+0x64f/0x710 [ldiskfs] Jan 3 18:16:34 service103 kernel: [] __down_read+0x7a/0x92 Jan 3 18:16:34 service103 kernel: [] down_read+0x11/0x13 Jan 3 18:16:34 service103 kernel: [] __dquot_free_space+0x3d/0x139 Jan 3 18:16:34 service103 kernel: [] dquot_free_space+0xb/0xd Jan 3 18:16:34 service103 kernel: [] ldiskfs_free_blocks+0xa3/0xc0 [ldiskfs] Jan 3 18:16:34 service103 kernel: [] ldiskfs_ext_truncate+0x50a/0xa80 [ldiskfs] Jan 3 18:16:34 service103 kernel: [] wake_up_bit+0x1e/0x23 Jan 3 18:16:35 service103 kernel: [] ldiskfs_truncate+0xb3/0x5c0 [ldiskfs] Jan 3 18:16:35 service103 kernel: [] kretprobe_trampoline+0x25/0x4b Jan 3 18:16:35 service103 kernel: [] __ldiskfs_handle_dirty_metadata+0xdb/0x110 [ldiskfs] Jan 3 18:16:35 service103 kernel: [] unmap_mapping_range+0x59/0x204 Jan 3 18:16:35 service103 kernel: [] ldiskfs_mark_iloc_dirty+0x4a5/0x540 [ldiskfs] Jan 3 18:16:35 service103 kernel: [] vmtruncate+0xa2/0xc9 Jan 3 18:16:35 service103 kernel: [] inode_setattr+0x22/0x104 Jan 3 18:16:35 service103 kernel: [] ldiskfs_setattr+0x2de/0x3a0 [ldiskfs] Jan 3 18:16:35 service103 kernel: [] fsfilt_ldiskfs_setattr+0x1a7/0x250 [fsfilt_ldiskfs] Jan 3 18:16:35 service103 kernel: [] filter_version_get_check+0x91/0x2a0 [obdfilter] Jan 3 18:16:36 service103 kernel: [] up_write+0x9/0xb Jan 3 18:16:36 service103 kernel: [] filter_destroy+0xd9b/0x1fb0 [obdfilter] Jan 3 18:16:36 service103 kernel: [] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] __ldlm_handle2lock+0x8c/0x360 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] ost_destroy+0x660/0x790 [ost] Jan 3 18:16:36 service103 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jan 3 18:16:36 service103 kernel: [] ost_handle+0x1556/0x55b0 [ost] Jan 3 18:16:37 service103 kernel: [] __rmqueue+0x44/0xc6 Jan 3 18:16:37 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:16:37 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:16:37 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:16:37 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 18:16:37 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:16:37 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:16:37 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:16:37 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:16:38 service103 kernel: Jan 3 18:16:38 service103 kernel: Pid: 10229, comm: ll_ost_io_48 Jan 3 18:16:38 service103 kernel: Jan 3 18:16:38 service103 kernel: Call Trace: Jan 3 18:16:38 service103 kernel: [] ldiskfs_mb_free_blocks+0x64f/0x710 [ldiskfs] Jan 3 18:16:38 service103 kernel: [] __down_read+0x7a/0x92 Jan 3 18:16:38 service103 kernel: [] down_read+0x11/0x13 Jan 3 18:16:39 service103 kernel: [] __dquot_free_space+0x3d/0x139 Jan 3 18:16:39 service103 kernel: [] dquot_free_space+0xb/0xd Jan 3 18:16:39 service103 kernel: [] ldiskfs_free_blocks+0xa3/0xc0 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] ldiskfs_ext_truncate+0x50a/0xa80 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] ldiskfs_truncate+0xb3/0x5c0 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] pagevec_lookup+0x17/0x1e Jan 3 18:16:39 service103 kernel: [] __ldiskfs_handle_dirty_metadata+0xdb/0x110 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] unmap_mapping_range+0x59/0x204 Jan 3 18:16:39 service103 kernel: [] ldiskfs_mark_iloc_dirty+0x4a5/0x540 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] vmtruncate+0xa2/0xc9 Jan 3 18:16:39 service103 kernel: [] inode_setattr+0x22/0x104 Jan 3 18:16:39 service103 kernel: [] ldiskfs_setattr+0x2de/0x3a0 [ldiskfs] Jan 3 18:16:39 service103 kernel: [] fsfilt_ldiskfs_setattr+0x1a7/0x250 [fsfilt_ldiskfs] Jan 3 18:16:39 service103 kernel: [] find_or_create_page+0x4b/0x72 Jan 3 18:16:39 service103 kernel: [] filter_setattr_internal+0xebb/0x1de0 [obdfilter] Jan 3 18:16:40 service103 kernel: [] kretprobe_trampoline+0x25/0x4b Jan 3 18:16:40 service103 kernel: [] __up_write+0xe4/0xf3 Jan 3 18:16:40 service103 kernel: [] filter_setattr+0x1c1/0x3b0 [obdfilter] Jan 3 18:16:40 service103 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Jan 3 18:16:40 service103 kernel: [] lustre_msg_buf+0x2c/0x90 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] lprocfs_counter_add+0x33/0x100 [lvfs] Jan 3 18:16:40 service103 kernel: [] ost_punch+0x9c8/0xce0 [ost] Jan 3 18:16:40 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] lustre_msg_check_version+0x1e/0x80 [ptlrpc] Jan 3 18:16:40 service103 kernel: [] ost_handle+0x3124/0x55b0 [ost] Jan 3 18:16:40 service103 kernel: [] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] ldlm_resource_get+0x1d9/0xa60 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] ldlm_resource_iterate+0x1ad/0x240 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:16:41 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:16:41 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:16:41 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:16:41 service103 kernel: Jan 3 18:16:41 service103 kernel: Lustre: Service thread pid 10220 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:41 service103 kernel: Lustre: Service thread pid 10228 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643385.10159 Jan 3 18:16:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643385.10202 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10205 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10053 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10112 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10178 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10147 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10137 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10119 Jan 3 18:16:42 service103 kernel: Lustre: Service thread pid 10062 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:42 service103 kernel: Lustre: Skipped 136 previous similar messages Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10141 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10054 Jan 3 18:16:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10061 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10062 Jan 3 18:16:43 service103 ntpd[8627]: kernel time sync enabled 0001 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10195 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10092 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10109 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643386.10069 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10088 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10094 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10201 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10170 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10145 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10111 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10082 Jan 3 18:16:43 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10066 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10077 Jan 3 18:16:44 service103 kernel: Lustre: Service thread pid 10153 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:44 service103 kernel: Lustre: Skipped 13 previous similar messages Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10146 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10056 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10076 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643387.10153 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10052 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10117 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10290 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10158 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10200 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10184 Jan 3 18:16:44 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10105 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10183 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10163 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10303 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10136 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10301 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643388.10101 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10300 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10175 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10086 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10296 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10098 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10289 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10132 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10167 Jan 3 18:16:45 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10283 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10192 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10211 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10083 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10182 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643389.10279 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10064 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10268 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10308 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10265 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10302 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10125 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10097 Jan 3 18:16:46 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10223 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10305 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10291 Jan 3 18:16:47 service103 kernel: Lustre: 19893:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff81094b783e00 already connecting Jan 3 18:16:47 service103 kernel: Lustre: 19893:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 4 previous similar messages Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10275 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10299 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10288 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643390.10270 Jan 3 18:16:47 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10058 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10280 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10282 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10166 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10263 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10285 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10096 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10287 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10150 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10217 Jan 3 18:16:48 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10281 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10258 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10104 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643391.10267 Jan 3 18:16:49 service103 kernel: Lustre: Service thread pid 10298 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:49 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10298 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10254 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10278 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10213 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10256 Jan 3 18:16:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10292 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10276 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10293 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10226 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10257 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10210 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10255 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10273 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10277 Jan 3 18:16:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643392.10271 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10272 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10242 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10253 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10208 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10297 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10148 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10073 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10164 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10059 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10249 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10252 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10245 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10295 Jan 3 18:16:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643393.10171 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10209 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10196 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10240 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10207 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10251 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10307 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10246 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10248 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10232 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10266 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10237 Jan 3 18:16:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10090 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10206 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643394.10294 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10264 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10250 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10247 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10260 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10236 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10071 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10286 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10244 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10235 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10269 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10227 Jan 3 18:16:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10131 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10128 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643395.10085 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10133 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10234 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10224 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10188 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10219 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10233 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10193 Jan 3 18:16:54 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10261 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10239 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10262 Jan 3 18:16:55 service103 kernel: Lustre: Service thread pid 10072 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:16:55 service103 kernel: Lustre: Skipped 8 previous similar messages Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10176 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10103 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10139 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643396.10072 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10241 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10222 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10189 Jan 3 18:16:55 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10231 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10218 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10274 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10238 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10118 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10157 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10177 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10122 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10284 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643397.10230 Jan 3 18:16:56 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10187 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10221 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10228 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10220 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10229 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10151 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10084 Jan 3 18:16:57 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10091 Jan 3 18:16:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10156 Jan 3 18:16:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10185 Jan 3 18:16:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10115 Jan 3 18:16:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10225 Jan 3 18:16:58 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643398.10186 Jan 3 18:16:58 service103 kernel: INFO: task ll_ost_00:10051 blocked for more than 120 seconds. Jan 3 18:16:58 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:16:58 service103 kernel: ll_ost_00 D ffff810c0cc88080 0 10051 1 10052 10034 (L-TLB) Jan 3 18:16:58 service103 kernel: ffff810c13905890 0000000000000046 0000000000000246 0000000000000000 Jan 3 18:16:58 service103 kernel: ffffffff800cef81 000000000000000a ffff810c0b781820 ffff810c0cc88080 Jan 3 18:16:58 service103 kernel: 0000013ceea64ea8 00000000000198a6 ffff810c0b781a08 0000000000020050 Jan 3 18:16:58 service103 kernel: Call Trace: Jan 3 18:16:59 service103 kernel: [] zone_statistics+0x6b/0x6e Jan 3 18:16:59 service103 kernel: [] list_add+0xc/0xe Jan 3 18:16:59 service103 kernel: [] :jbd2:start_this_handle+0x301/0x3cb Jan 3 18:16:59 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:16:59 service103 kernel: [] :jbd2:jbd2_journal_start+0xab/0xdf Jan 3 18:16:59 service103 kernel: [] :ldiskfs:ldiskfs_journal_start_sb+0x55/0xa0 Jan 3 18:16:59 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590 Jan 3 18:16:59 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:16:59 service103 kernel: [] :lvfs:push_ctxt+0x370/0x380 Jan 3 18:16:59 service103 kernel: [] :obdfilter:filter_client_add+0x508/0xc30 Jan 3 18:16:59 service103 kernel: [] proc_register+0x7d/0x115 Jan 3 18:16:59 service103 kernel: [] :obdclass:lprocfs_register_stats+0x50/0x80 Jan 3 18:16:59 service103 kernel: [] :obdfilter:filter_export_stats_init+0x539/0x650 Jan 3 18:16:59 service103 kernel: [] :obdfilter:filter_connect+0x535/0x8c0 Jan 3 18:16:59 service103 kernel: [] :ptlrpc:lustre_msg_add_op_flags+0x47/0x120 Jan 3 18:17:00 service103 kernel: [] :ost:ost_handle+0x0/0x55b0 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:target_handle_connect+0x21c6/0x2e80 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Jan 3 18:17:00 service103 kernel: [] :ost:ost_handle+0x8af/0x55b0 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:00 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:00 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:00 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:01 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:01 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:01 service103 kernel: Jan 3 18:17:01 service103 kernel: INFO: task ll_ost_01:10052 blocked for more than 120 seconds. Jan 3 18:17:01 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:01 service103 kernel: ll_ost_01 D ffff81000900caa0 0 10052 1 10053 10051 (L-TLB) Jan 3 18:17:01 service103 kernel: ffff810c13907a50 0000000000000046 ffff810c243bb080 0000000000000000 Jan 3 18:17:01 service103 kernel: ffff810c20be4bc0 000000000000000a ffff810c243bb080 ffff81012af11100 Jan 3 18:17:01 service103 kernel: 000001356efab78f 0000000000008b38 ffff810c243bb268 0000000100000004 Jan 3 18:17:01 service103 kernel: Call Trace: Jan 3 18:17:01 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:17:01 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:17:02 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:17:02 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+0x96e/0xf60 Jan 3 18:17:02 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:02 service103 kernel: [] :lquota:filter_quota_ctl+0x272/0xbf0 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Jan 3 18:17:02 service103 kernel: [] :ost:ost_handle+0x478b/0x55b0 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:02 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:02 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:02 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:02 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:02 service103 kernel: Jan 3 18:17:03 service103 kernel: INFO: task ll_ost_02:10053 blocked for more than 120 seconds. Jan 3 18:17:03 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:03 service103 kernel: ll_ost_02 D ffff81000901d7a0 0 10053 1 10054 10052 (L-TLB) Jan 3 18:17:03 service103 kernel: ffff810c13911a50 0000000000000046 0005000000000000 ffff810a6a622400 Jan 3 18:17:03 service103 kernel: 000500000a973a0f 000000000000000a ffff810c3b9c7040 ffff81012af80100 Jan 3 18:17:03 service103 kernel: 00000135ba199931 0000000000002238 ffff810c3b9c7228 0000000300000004 Jan 3 18:17:03 service103 kernel: Call Trace: Jan 3 18:17:03 service103 kernel: [] :lnet:LNetPut+0x730/0x840 Jan 3 18:17:03 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:17:03 service103 kernel: [] :ptlrpc:ptl_send_buf+0x3f3/0x5b0 Jan 3 18:17:03 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:17:03 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:17:03 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+0x96e/0xf60 Jan 3 18:17:03 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:03 service103 kernel: [] :lquota:filter_quota_ctl+0x272/0xbf0 Jan 3 18:17:04 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 18:17:04 service103 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Jan 3 18:17:04 service103 kernel: [] :ost:ost_handle+0x478b/0x55b0 Jan 3 18:17:04 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:04 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:04 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 18:17:04 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:04 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:04 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:04 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:04 service103 kernel: Jan 3 18:17:04 service103 kernel: INFO: task ll_ost_03:10054 blocked for more than 120 seconds. Jan 3 18:17:04 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:04 service103 kernel: ll_ost_03 D ffff810c1237b860 0 10054 1 10055 10053 (L-TLB) Jan 3 18:17:05 service103 kernel: ffff810c13913a50 0000000000000046 0005000000000000 ffff810b371cb400 Jan 3 18:17:05 service103 kernel: 000500000a970c42 000000000000000a ffff810c12be1100 ffff810c1237b860 Jan 3 18:17:05 service103 kernel: 00000135dd4ee1e1 000000000000c217 ffff810c12be12e8 0000000100000004 Jan 3 18:17:05 service103 kernel: Call Trace: Jan 3 18:17:05 service103 kernel: [] :lnet:LNetPut+0x730/0x840 Jan 3 18:17:05 service103 kernel: [] :lnet:LNetMDBind+0x301/0x450 Jan 3 18:17:05 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:17:05 service103 kernel: [] :ptlrpc:ptl_send_buf+0x3f3/0x5b0 Jan 3 18:17:05 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:17:05 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:17:05 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+0x96e/0xf60 Jan 3 18:17:05 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:05 service103 kernel: [] :lquota:filter_quota_ctl+0x272/0xbf0 Jan 3 18:17:05 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 18:17:06 service103 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Jan 3 18:17:06 service103 kernel: [] :ost:ost_handle+0x478b/0x55b0 Jan 3 18:17:06 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:17:06 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:06 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:06 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:06 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:06 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:06 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:06 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:06 service103 kernel: Jan 3 18:17:06 service103 kernel: INFO: task ll_ost_04:10055 blocked for more than 120 seconds. Jan 3 18:17:06 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:06 service103 kernel: ll_ost_04 D ffff810009025e20 0 10055 1 10056 10054 (L-TLB) Jan 3 18:17:07 service103 kernel: ffff810c1391d890 0000000000000046 ffff810c18061558 000500000a972103 Jan 3 18:17:07 service103 kernel: ffff810c1391d8a0 000000000000000a ffff810c14ec6860 ffff81012afb7080 Jan 3 18:17:07 service103 kernel: 0000013cae9dc738 0000000000014a61 ffff810c14ec6a48 00000004ffffffff Jan 3 18:17:07 service103 kernel: Call Trace: Jan 3 18:17:07 service103 kernel: [] vsnprintf+0x3f8/0x627 Jan 3 18:17:07 service103 kernel: [] :jbd2:start_this_handle+0x301/0x3cb Jan 3 18:17:07 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:17:07 service103 kernel: [] :jbd2:jbd2_journal_start+0xab/0xdf Jan 3 18:17:07 service103 kernel: [] :ldiskfs:ldiskfs_journal_start_sb+0x55/0xa0 Jan 3 18:17:07 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590 Jan 3 18:17:07 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:07 service103 kernel: [] :lvfs:push_ctxt+0x370/0x380 Jan 3 18:17:07 service103 kernel: [] :obdfilter:filter_client_add+0x508/0xc30 Jan 3 18:17:07 service103 kernel: [] proc_register+0x7d/0x115 Jan 3 18:17:07 service103 kernel: [] :obdclass:lprocfs_register_stats+0x50/0x80 Jan 3 18:17:08 service103 kernel: [] :obdfilter:filter_export_stats_init+0x539/0x650 Jan 3 18:17:08 service103 kernel: [] :obdfilter:filter_connect+0x535/0x8c0 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:lustre_msg_add_op_flags+0x47/0x120 Jan 3 18:17:08 service103 kernel: [] :ost:ost_handle+0x0/0x55b0 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:target_handle_connect+0x21c6/0x2e80 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Jan 3 18:17:08 service103 kernel: [] :ost:ost_handle+0x8af/0x55b0 Jan 3 18:17:08 service103 kernel: [] __next_cpu+0x19/0x28 Jan 3 18:17:08 service103 kernel: [] smp_send_reschedule+0x4e/0x53 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:08 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:08 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:09 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:09 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:09 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:09 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:09 service103 kernel: Jan 3 18:17:09 service103 kernel: INFO: task ll_ost_05:10056 blocked for more than 120 seconds. Jan 3 18:17:09 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:09 service103 kernel: ll_ost_05 D ffff810c0d45d860 0 10056 1 10057 10055 (L-TLB) Jan 3 18:17:09 service103 kernel: ffff810c13925a50 0000000000000046 0005000000000000 ffff810b3204fa00 Jan 3 18:17:09 service103 kernel: 000500000a970dcd 000000000000000a ffff810c14ec6100 ffff810c0d45d860 Jan 3 18:17:09 service103 kernel: 000001362839d5d3 0000000000003893 ffff810c14ec62e8 0000000400000004 Jan 3 18:17:09 service103 kernel: Call Trace: Jan 3 18:17:09 service103 kernel: [] :lnet:LNetPut+0x730/0x840 Jan 3 18:17:09 service103 kernel: [] :lnet:LNetMDBind+0x301/0x450 Jan 3 18:17:10 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:17:10 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:17:10 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:17:10 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+0x96e/0xf60 Jan 3 18:17:10 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:10 service103 kernel: [] :lquota:filter_quota_ctl+0x272/0xbf0 Jan 3 18:17:10 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 18:17:10 service103 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Jan 3 18:17:10 service103 kernel: [] :ost:ost_handle+0x478b/0x55b0 Jan 3 18:17:10 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:17:10 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:10 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:10 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 18:17:10 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:10 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:11 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:11 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:11 service103 kernel: Jan 3 18:17:11 service103 kernel: INFO: task ll_ost_06:10057 blocked for more than 120 seconds. Jan 3 18:17:11 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:11 service103 kernel: ll_ost_06 D ffff810009025e20 0 10057 1 10058 10056 (L-TLB) Jan 3 18:17:11 service103 kernel: ffff810c13927890 0000000000000046 ffff810ad5b02958 000500000a973492 Jan 3 18:17:11 service103 kernel: ffff810c139278a0 000000000000000a ffff810c14ec97a0 ffff81012afb7080 Jan 3 18:17:11 service103 kernel: 000001457b51bbbb 0000000000015290 ffff810c14ec9988 00000004ffffffff Jan 3 18:17:11 service103 kernel: Call Trace: Jan 3 18:17:11 service103 kernel: [] vsnprintf+0x3f8/0x627 Jan 3 18:17:11 service103 kernel: [] :jbd2:start_this_handle+0x301/0x3cb Jan 3 18:17:11 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:17:11 service103 kernel: [] :jbd2:jbd2_journal_start+0xab/0xdf Jan 3 18:17:12 service103 kernel: [] :ldiskfs:ldiskfs_journal_start_sb+0x55/0xa0 Jan 3 18:17:12 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590 Jan 3 18:17:12 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:12 service103 kernel: [] :lvfs:push_ctxt+0x370/0x380 Jan 3 18:17:12 service103 kernel: [] :obdfilter:filter_client_add+0x508/0xc30 Jan 3 18:17:12 service103 kernel: [] proc_register+0x7d/0x115 Jan 3 18:17:12 service103 kernel: [] :obdclass:lprocfs_register_stats+0x50/0x80 Jan 3 18:17:12 service103 kernel: [] :obdfilter:filter_export_stats_init+0x539/0x650 Jan 3 18:17:12 service103 kernel: [] :obdfilter:filter_connect+0x535/0x8c0 Jan 3 18:17:12 service103 kernel: [] :ptlrpc:lustre_msg_add_op_flags+0x47/0x120 Jan 3 18:17:12 service103 kernel: [] :ost:ost_handle+0x0/0x55b0 Jan 3 18:17:12 service103 kernel: [] :ptlrpc:target_handle_connect+0x21c6/0x2e80 Jan 3 18:17:12 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 18:17:12 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 18:17:12 service103 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Jan 3 18:17:13 service103 kernel: [] :ost:ost_handle+0x8af/0x55b0 Jan 3 18:17:13 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:13 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:13 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:13 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:13 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:13 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:13 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:13 service103 kernel: Jan 3 18:17:13 service103 kernel: INFO: task ll_ost_07:10058 blocked for more than 120 seconds. Jan 3 18:17:13 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:13 service103 kernel: ll_ost_07 D ffff810c23d96860 0 10058 1 10059 10057 (L-TLB) Jan 3 18:17:13 service103 kernel: ffff810c13931a50 0000000000000046 0005000000000000 ffff810920165600 Jan 3 18:17:13 service103 kernel: 000500000a9730bc 000000000000000a ffff810c14ec9040 ffff810c23d96860 Jan 3 18:17:14 service103 kernel: 000001352d54476e 000000000000fd29 ffff810c14ec9228 0000000700000004 Jan 3 18:17:14 service103 kernel: Call Trace: Jan 3 18:17:14 service103 kernel: [] :lnet:LNetPut+0x730/0x840 Jan 3 18:17:14 service103 kernel: [] :lnet:LNetMDBind+0x301/0x450 Jan 3 18:17:14 service103 kernel: [] __mutex_lock_slowpath+0x60/0x9b Jan 3 18:17:14 service103 kernel: [] .text.lock.mutex+0xf/0x14 Jan 3 18:17:14 service103 kernel: [] vfs_get_dqblk+0x23/0xcf Jan 3 18:17:14 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+0x96e/0xf60 Jan 3 18:17:14 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:14 service103 kernel: [] :lquota:filter_quota_ctl+0x272/0xbf0 Jan 3 18:17:14 service103 kernel: [] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950 Jan 3 18:17:14 service103 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Jan 3 18:17:14 service103 kernel: [] :ost:ost_handle+0x478b/0x55b0 Jan 3 18:17:14 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:17:15 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:15 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:15 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:15 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:15 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:15 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:15 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:15 service103 kernel: Jan 3 18:17:15 service103 kernel: INFO: task ll_ost_08:10059 blocked for more than 120 seconds. Jan 3 18:17:15 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:15 service103 kernel: ll_ost_08 D ffff810c0d45a0c0 0 10059 1 10060 10058 (L-TLB) Jan 3 18:17:15 service103 kernel: ffff810c13939890 0000000000000046 ffff8109d59a5d58 000500000a97055b Jan 3 18:17:15 service103 kernel: ffff810c139398a0 000000000000000a ffff810c14ecc7e0 ffff810c0d45a0c0 Jan 3 18:17:15 service103 kernel: 000001377bb00b7f 000000000001fab1 ffff810c14ecc9c8 00000000ffffffff Jan 3 18:17:16 service103 kernel: Call Trace: Jan 3 18:17:16 service103 kernel: [] vsnprintf+0x3f8/0x627 Jan 3 18:17:16 service103 kernel: [] list_add+0xc/0xe Jan 3 18:17:16 service103 kernel: [] :jbd2:start_this_handle+0x301/0x3cb Jan 3 18:17:16 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:17:16 service103 kernel: [] :jbd2:jbd2_journal_start+0xab/0xdf Jan 3 18:17:16 service103 kernel: [] :ldiskfs:ldiskfs_journal_start_sb+0x55/0xa0 Jan 3 18:17:16 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590 Jan 3 18:17:16 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:16 service103 kernel: [] :lvfs:push_ctxt+0x370/0x380 Jan 3 18:17:16 service103 kernel: [] :obdfilter:filter_client_add+0x508/0xc30 Jan 3 18:17:16 service103 kernel: [] proc_register+0x7d/0x115 Jan 3 18:17:16 service103 kernel: [] :obdclass:lprocfs_register_stats+0x50/0x80 Jan 3 18:17:16 service103 kernel: [] :obdfilter:filter_export_stats_init+0x539/0x650 Jan 3 18:17:16 service103 kernel: [] :obdfilter:filter_connect+0x535/0x8c0 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:lustre_msg_add_op_flags+0x47/0x120 Jan 3 18:17:17 service103 kernel: [] :ost:ost_handle+0x0/0x55b0 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:target_handle_connect+0x21c6/0x2e80 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Jan 3 18:17:17 service103 kernel: [] :ost:ost_handle+0x8af/0x55b0 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:17 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:17 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:17 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:17 service103 kernel: Jan 3 18:17:18 service103 kernel: INFO: task ll_ost_09:10060 blocked for more than 120 seconds. Jan 3 18:17:18 service103 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 3 18:17:18 service103 kernel: ll_ost_09 D ffff810c113c0080 0 10060 1 10061 10059 (L-TLB) Jan 3 18:17:18 service103 kernel: ffff810c1393b890 0000000000000046 ffff8109fbc68158 000500000a973492 Jan 3 18:17:18 service103 kernel: ffff810c1393b8a0 000000000000000a ffff810c14ecc080 ffff810c113c0080 Jan 3 18:17:18 service103 kernel: 000001457b58af3c 0000000000014833 ffff810c14ecc268 00000001ffffffff Jan 3 18:17:18 service103 kernel: Call Trace: Jan 3 18:17:18 service103 kernel: [] vsnprintf+0x3f8/0x627 Jan 3 18:17:18 service103 kernel: [] list_add+0xc/0xe Jan 3 18:17:18 service103 kernel: [] :jbd2:start_this_handle+0x301/0x3cb Jan 3 18:17:18 service103 kernel: Lustre: Service thread pid 10243 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:17:18 service103 kernel: Lustre: Skipped 11 previous similar messages Jan 3 18:17:18 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643406.10243 Jan 3 18:17:18 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:17:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643406.10306 Jan 3 18:17:19 service103 kernel: [] :jbd2:jbd2_journal_start+0xab/0xdf Jan 3 18:17:19 service103 kernel: [] :ldiskfs:ldiskfs_journal_start_sb+0x55/0xa0 Jan 3 18:17:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643406.10190 Jan 3 18:17:19 service103 kernel: [] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590 Jan 3 18:17:19 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:17:19 service103 kernel: [] :lvfs:push_ctxt+0x370/0x380 Jan 3 18:17:19 service103 kernel: [] :obdfilter:filter_client_add+0x508/0xc30 Jan 3 18:17:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.10214 Jan 3 18:17:19 service103 kernel: [] proc_register+0x7d/0x115 Jan 3 18:17:19 service103 kernel: [] :obdclass:lprocfs_register_stats+0x50/0x80 Jan 3 18:17:19 service103 kernel: [] :obdfilter:filter_export_stats_init+0x539/0x650 Jan 3 18:17:19 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18885 Jan 3 18:17:19 service103 kernel: [] :obdfilter:filter_connect+0x535/0x8c0 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:lustre_msg_add_op_flags+0x47/0x120 Jan 3 18:17:20 service103 kernel: [] :ost:ost_handle+0x0/0x55b0 Jan 3 18:17:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18913 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:target_handle_connect+0x21c6/0x2e80 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600 Jan 3 18:17:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18912 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Jan 3 18:17:20 service103 kernel: [] :ost:ost_handle+0x8af/0x55b0 Jan 3 18:17:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18911 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Jan 3 18:17:20 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:17:20 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.10065 Jan 3 18:17:20 service103 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Jan 3 18:17:21 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:17:21 service103 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.10074 Jan 3 18:17:21 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:17:21 service103 kernel: Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.10110 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18921 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18920 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18919 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18918 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18917 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18905 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.10144 Jan 3 18:17:21 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643407.18906 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18904 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18903 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18902 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.10070 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.10123 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18901 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18898 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18900 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18899 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18897 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18896 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18895 Jan 3 18:17:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18894 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643408.18892 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18891 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.10198 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18890 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18889 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18888 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18887 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18886 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18884 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18883 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18882 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18881 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.18880 Jan 3 18:17:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.10203 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643409.10113 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.18879 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.18878 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.18877 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.18876 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.18875 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10197 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10216 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10304 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10194 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10204 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10191 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10181 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10212 Jan 3 18:17:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643410.10215 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643411.18953 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18952 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18951 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18950 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18948 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18949 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643412.18947 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18986 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18985 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18984 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18992 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18991 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18990 Jan 3 18:17:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18989 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18988 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18987 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18983 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643414.18982 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643415.18981 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643415.19001 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643415.10089 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643415.10078 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643415.10152 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.10055 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.10138 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19008 Jan 3 18:17:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19007 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19006 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19005 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19004 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19018 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19024 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19023 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19022 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.10173 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.19031 Jan 3 18:17:27 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.10161 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643416.10149 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.10142 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.10051 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19032 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19026 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19030 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19021 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19020 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19019 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19017 Jan 3 18:17:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19016 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19015 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19014 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19013 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643417.19003 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643418.19000 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643418.18999 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643418.18998 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643418.18997 Jan 3 18:17:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643418.18996 Jan 3 18:17:29 service103 kernel: Lustre: nbp6-OST005a: haven't heard from client ac1979d4-e0c4-6fab-e981-927527aea406 (at (no nid)) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 18:17:29 service103 kernel: Lustre: nbp6-OST005a: haven't heard from client f2ebc8aa-399d-39a6-b6bf-2336ea784f62 (at (no nid)) in 225 seconds. I think it's dead, and I am evicting it. Jan 3 18:17:29 service103 kernel: Lustre: 19874:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff8109b9092e00 already connecting Jan 3 18:17:29 service103 kernel: Lustre: 19874:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 5 previous similar messages Jan 3 18:17:29 service103 kernel: Lustre: 20589:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff810999693a00 already connecting Jan 3 18:17:30 service103 kernel: Lustre: 20589:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 15 previous similar messages Jan 3 18:17:30 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client ee954ebc-9c00-aec7-44e8-db5092495588 (at (no nid)) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 18:17:30 service103 kernel: Lustre: Skipped 45 previous similar messages Jan 3 18:17:30 service103 kernel: Lustre: Service thread pid 10093 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:17:30 service103 kernel: Lustre: Skipped 117 previous similar messages Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10093 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10143 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10087 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10106 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10075 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643446.10114 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10067 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10079 Jan 3 18:17:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10124 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10130 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10174 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10168 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10108 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10063 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10107 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10165 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10140 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10169 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643447.10068 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10162 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10080 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10099 Jan 3 18:17:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10127 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10120 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10129 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10172 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10134 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643448.10154 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643449.10116 Jan 3 18:17:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643449.10121 Jan 3 18:17:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643453.10057 Jan 3 18:17:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643453.10060 Jan 3 18:17:37 service103 kernel: Lustre: 20629:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff810bc277f400 already connecting Jan 3 18:17:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643461.10126 Jan 3 18:17:41 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643461.10102 Jan 3 18:17:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643469.10095 Jan 3 18:17:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643469.10160 Jan 3 18:17:49 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643469.19273 Jan 3 18:17:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643470.10100 Jan 3 18:18:03 service103 kernel: LustreError: 18683:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1325643183, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-nbp6-OST0012_UUID lock: ffff810c1cc52c00/0x69beea547f8bb5d7 lrc: 3/0,1 mode: --/PW res: 278724/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 18683 timeout 0 Jan 3 18:18:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643483.18683 Jan 3 18:18:03 service103 kernel: LustreError: 18688:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1325643183, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-nbp6-OST0062_UUID lock: ffff810c1d319600/0x69beea547f8bb6e8 lrc: 3/0,1 mode: --/PW res: 274710/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 18688 timeout 0 Jan 3 18:18:05 service103 kernel: LustreError: 18732:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1325643185, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-nbp6-OST0072_UUID lock: ffff81091dadb200/0x69beea547f8bb9d5 lrc: 3/0,1 mode: --/PW res: 278650/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 18732 timeout 0 Jan 3 18:18:05 service103 kernel: LustreError: 18732:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jan 3 18:18:16 service103 kernel: Lustre: nbp6-OST0012: haven't heard from client 12fb0d8a-0624-9476-0f6f-9b0449cb9cf1 (at (no nid)) in 214 seconds. I think it's dead, and I am evicting it. Jan 3 18:18:16 service103 kernel: Lustre: Skipped 21 previous similar messages Jan 3 18:18:25 service103 kernel: Lustre: 20582:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff810978551600 already connecting Jan 3 18:18:25 service103 kernel: Lustre: 20582:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 3 previous similar messages Jan 3 18:18:27 service103 kernel: LustreError: 18910:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1325643207, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-nbp6-OST0002_UUID lock: ffff8109c3fb1200/0x69beea547f8bc48d lrc: 3/0,1 mode: --/PW res: 274905/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 18910 timeout 0 Jan 3 18:18:38 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 75b74635-7b6f-146d-5623-2841ff309de3 (at (no nid)) in 170 seconds. I think it's dead, and I am evicting it. Jan 3 18:18:38 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:19:00 service103 kernel: Lustre: Service thread pid 19244 was inactive for 258.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:19:00 service103 kernel: Lustre: Skipped 37 previous similar messages Jan 3 18:19:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643540.19244 Jan 3 18:19:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643540.10081 Jan 3 18:19:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643540.19274 Jan 3 18:19:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643540.19276 Jan 3 18:19:00 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643540.19275 Jan 3 18:19:32 service103 kernel: Lustre: nbp6-OST005a: haven't heard from client 75b74635-7b6f-146d-5623-2841ff309de3 (at (no nid)) in 224 seconds. I think it's dead, and I am evicting it. Jan 3 18:19:54 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client ac1979d4-e0c4-6fab-e981-927527aea406 (at (no nid)) in 226 seconds. I think it's dead, and I am evicting it. Jan 3 18:19:54 service103 kernel: Lustre: Skipped 8 previous similar messages Jan 3 18:19:57 service103 kernel: Lustre: 21030:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0062: exp ffff81092afd5000 already connecting Jan 3 18:19:57 service103 kernel: Lustre: 21030:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 11 previous similar messages Jan 3 18:20:06 service103 kernel: Lustre: Service thread pid 10155 was inactive for 258.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:20:06 service103 kernel: Lustre: Skipped 4 previous similar messages Jan 3 18:20:06 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643606.19318 Jan 3 18:20:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643607.19322 Jan 3 18:20:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643607.19319 Jan 3 18:20:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643607.10135 Jan 3 18:20:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643607.10155 Jan 3 18:20:07 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643607.19533 Jan 3 18:20:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643609.19536 Jan 3 18:20:26 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643626.19325 Jan 3 18:20:28 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643628.19532 Jan 3 18:20:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643629.19535 Jan 3 18:20:30 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643630.19858 Jan 3 18:20:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643631.19866 Jan 3 18:20:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643637.19321 Jan 3 18:20:48 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client 6c6fc931-ec9b-e6fd-d32f-a1c928bba5bd (at (no nid)) in 226 seconds. I think it's dead, and I am evicting it. Jan 3 18:20:48 service103 kernel: Lustre: Skipped 5 previous similar messages Jan 3 18:21:38 service103 kernel: LustreError: 21410:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114) req@ffff810c01f25400 x1390035605698940/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1325643798 ref 1 fl Interpret:/0/0 rc -114/0 Jan 3 18:21:38 service103 kernel: LustreError: 21410:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 96 previous similar messages Jan 3 18:22:04 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client ac1979d4-e0c4-6fab-e981-927527aea406 (at (no nid)) in 181 seconds. I think it's dead, and I am evicting it. Jan 3 18:22:04 service103 kernel: Lustre: Skipped 18 previous similar messages Jan 3 18:22:20 service103 kernel: Lustre: 21690:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0002: exp ffff81092c985e00 already connecting Jan 3 18:22:20 service103 kernel: Lustre: 21689:0:(ldlm_lib.c:803:target_handle_connect()) nbp6-OST0012: exp ffff81092cfd3c00 already connecting Jan 3 18:22:20 service103 kernel: Lustre: 21689:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 39 previous similar messages Jan 3 18:22:20 service103 kernel: Lustre: 21690:0:(ldlm_lib.c:803:target_handle_connect()) Skipped 2 previous similar messages Jan 3 18:23:19 service103 pcp-pmie[8185]: High 1-minute load average 475load@service103 Jan 3 18:23:20 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client 2c0ec8cc-dbae-68d9-234d-ed9e92b7d532 (at (no nid)) in 204 seconds. I think it's dead, and I am evicting it. Jan 3 18:23:20 service103 kernel: Lustre: Skipped 45 previous similar messages Jan 3 18:23:40 service103 kernel: Lustre: Service thread pid 19027 was inactive for 434.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:23:40 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 18:23:40 service103 kernel: Pid: 19027, comm: ll_ost_io_213 Jan 3 18:23:40 service103 kernel: Jan 3 18:23:40 service103 kernel: Call Trace: Jan 3 18:23:40 service103 kernel: [] ldiskfs_get_blocks+0xcf/0x210 [ldiskfs] Jan 3 18:23:40 service103 kernel: [] quota_is_set+0xf8/0x230 [lquota] Jan 3 18:23:40 service103 kernel: [] __down_write_nested+0x7a/0x92 Jan 3 18:23:40 service103 kernel: [] __down_write+0xb/0xd Jan 3 18:23:40 service103 kernel: [] down_write+0x11/0x13 Jan 3 18:23:40 service103 kernel: [] dquot_initialize+0x2e/0xac Jan 3 18:23:40 service103 kernel: [] filter_commitrw_write+0x93c/0x2dc0 [obdfilter] Jan 3 18:23:40 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:23:40 service103 kernel: [] lnet_ni_send+0x96/0xc0 [lnet] Jan 3 18:23:40 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:23:40 service103 kernel: [] filter_commitrw+0x65/0x2c0 [obdfilter] Jan 3 18:23:40 service103 kernel: [] ost_brw_write+0x1c99/0x2480 [ost] Jan 3 18:23:41 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] lustre_msg_set_last_committed+0x45/0x120 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] ptlrpc_reply+0xb/0x10 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] default_wake_function+0x0/0xf Jan 3 18:23:41 service103 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Jan 3 18:23:41 service103 kernel: [] lock_handle_addref+0x5/0x10 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 3 18:23:41 service103 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:23:41 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:23:41 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:23:42 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:23:42 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:23:42 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:23:42 service103 kernel: Jan 3 18:23:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643820.19027 Jan 3 18:25:27 service103 kernel: Lustre: Service thread pid 19897 was inactive for 506.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:25:27 service103 kernel: Pid: 19897, comm: ll_ost_149 Jan 3 18:25:27 service103 kernel: Jan 3 18:25:27 service103 kernel: Call Trace: Jan 3 18:25:27 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:25:27 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:25:27 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:25:27 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:25:27 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:25:27 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:25:27 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:25:27 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:25:27 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:25:27 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:25:27 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:25:27 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:25:28 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:25:28 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:25:28 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:25:28 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:25:28 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:25:29 service103 kernel: Jan 3 18:25:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643927.19897 Jan 3 18:25:29 service103 kernel: Pid: 19550, comm: ll_ost_142 Jan 3 18:25:29 service103 kernel: Jan 3 18:25:29 service103 kernel: Call Trace: Jan 3 18:25:29 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:25:29 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:25:29 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:25:30 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:25:30 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:25:30 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:25:30 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:25:30 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:25:30 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:25:30 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:25:30 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:25:30 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:25:30 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:25:30 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:25:30 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:25:30 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:25:31 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:25:31 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:25:31 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:25:31 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:25:31 service103 kernel: Jan 3 18:25:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643928.19550 Jan 3 18:25:31 service103 kernel: Pid: 19899, comm: ll_ost_150 Jan 3 18:25:31 service103 kernel: Jan 3 18:25:31 service103 kernel: Call Trace: Jan 3 18:25:32 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:25:32 service103 kernel: [] list_add+0xc/0xe Jan 3 18:25:32 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:25:32 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:25:32 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:25:32 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:25:32 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:25:32 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:25:32 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:25:32 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:25:32 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:25:32 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:25:32 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:25:33 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:25:33 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:25:33 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] utrace_report_death+0x1e8/0x1f5 Jan 3 18:25:33 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:25:33 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:25:33 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:25:34 service103 kernel: Jan 3 18:25:34 service103 kernel: Pid: 20301, comm: ll_ost_152 Jan 3 18:25:34 service103 kernel: Jan 3 18:25:34 service103 kernel: Call Trace: Jan 3 18:25:34 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:25:34 service103 kernel: [] list_add+0xc/0xe Jan 3 18:25:34 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:25:34 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:25:34 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:25:34 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:25:34 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:25:34 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:25:34 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:25:35 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:25:35 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:25:35 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:25:35 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:25:35 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:25:35 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:25:35 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:25:35 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:25:36 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:25:36 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:25:36 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:25:36 service103 kernel: Jan 3 18:25:36 service103 kernel: Lustre: Service thread pid 19865 was inactive for 506.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 3 18:25:36 service103 kernel: Lustre: Skipped 12 previous similar messages Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20579 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20581 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20578 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20580 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20583 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.19825 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.19906 Jan 3 18:25:36 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.20302 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.19874 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643929.19865 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643930.20301 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643930.19899 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643930.20586 Jan 3 18:25:37 service103 kernel: Lustre: 19033:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jan 3 18:25:37 service103 kernel: req@ffff8109f987d000 x1387572511431225/t0 o6->7abfce6f-1165-363a-a0e1-b95a1a28ea63@NET_0x500000a9719d4_UUID:0/0 lens 512/400 e 0 to 0 dl 1325643938 ref 2 fl Interpret:/2/0 rc 0/0 Jan 3 18:25:37 service103 kernel: Lustre: 19033:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jan 3 18:25:37 service103 kernel: req@ffff810c19e33850 x1387572511431138/t0 o6->7abfce6f-1165-363a-a0e1-b95a1a28ea63@NET_0x500000a9719d4_UUID:0/0 lens 512/400 e 0 to 0 dl 1325643938 ref 2 fl Interpret:/2/0 rc 0/0 Jan 3 18:25:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643934.19893 Jan 3 18:25:42 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643942.20585 Jan 3 18:25:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643950.20623 Jan 3 18:25:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643950.20587 Jan 3 18:25:50 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643950.20589 Jan 3 18:25:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643950.20627 Jan 3 18:25:57 service103 kernel: Lustre: 18893:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jan 3 18:25:57 service103 kernel: req@ffff8109f9888800 x1390004811972762/t0 o4->d4d77a0e-9bb3-4873-89c2-8616f410f774@NET_0x500000a97353e_UUID:0/0 lens 448/416 e 0 to 0 dl 1325643962 ref 2 fl Interpret:/2/0 rc 0/0 Jan 3 18:25:57 service103 kernel: Lustre: 18893:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Jan 3 18:26:03 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325643963.20592 Jan 3 18:26:07 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client ee954ebc-9c00-aec7-44e8-db5092495588 (at (no nid)) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 18:26:07 service103 kernel: Lustre: Skipped 20 previous similar messages Jan 3 18:26:46 service103 kernel: Lustre: 19526:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 18:26:46 service103 kernel: req@ffff8109437be000 x1390004815143663/t0 o4->9afaba0f-9f6b-c27e-0f9b-2433e0718845@NET_0x500000a973582_UUID:0/0 lens 448/416 e 2 to 0 dl 1325644011 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 18:26:46 service103 kernel: Lustre: 19526:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages Jan 3 18:26:50 service103 kernel: Lustre: 19028:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 18:26:50 service103 kernel: req@ffff8109711c8000 x1390004820414858/t0 o4->94082c47-9271-0e79-8444-c99bb0a4505d@NET_0x500000a97357e_UUID:0/0 lens 448/416 e 2 to 0 dl 1325644015 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 18:26:50 service103 kernel: Lustre: 19028:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 19 previous similar messages Jan 3 18:27:09 service103 kernel: Lustre: 10259:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 18:27:09 service103 kernel: req@ffff8109a3481000 x1390004818284793/t0 o4->cfb92187-6de2-430c-2490-9b585d829aca@NET_0x500000a97357b_UUID:0/0 lens 448/416 e 2 to 0 dl 1325644034 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 18:27:09 service103 kernel: Lustre: 10259:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Jan 3 18:27:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644029.20729 Jan 3 18:27:19 service103 kernel: Lustre: 19029:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-223), not sending early reply Jan 3 18:27:19 service103 kernel: req@ffff8109f7ce0400 x1390004820403147/t0 o4->a30eccc5-95f4-5476-6077-7042d2a84fa8@NET_0x500000a973586_UUID:0/0 lens 448/416 e 2 to 0 dl 1325644044 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 18:27:19 service103 kernel: Lustre: 19029:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Jan 3 18:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644049.20630 Jan 3 18:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644049.20822 Jan 3 18:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644049.20629 Jan 3 18:27:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644049.20628 Jan 3 18:27:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644051.20582 Jan 3 18:27:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644051.20584 Jan 3 18:27:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644051.20868 Jan 3 18:27:31 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644051.20923 Jan 3 18:27:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644052.20925 Jan 3 18:27:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644052.20944 Jan 3 18:27:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644052.20918 Jan 3 18:27:32 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644052.20941 Jan 3 18:27:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644053.20943 Jan 3 18:27:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644053.20947 Jan 3 18:27:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644053.20945 Jan 3 18:27:33 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644053.20924 Jan 3 18:27:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644054.20940 Jan 3 18:27:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644054.20948 Jan 3 18:27:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644054.20942 Jan 3 18:27:34 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644054.20951 Jan 3 18:27:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644071.20953 Jan 3 18:27:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644071.20954 Jan 3 18:27:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644071.20955 Jan 3 18:27:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644071.20952 Jan 3 18:27:51 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644071.20949 Jan 3 18:27:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644072.20992 Jan 3 18:27:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644072.20994 Jan 3 18:27:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644072.20989 Jan 3 18:27:52 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644072.20991 Jan 3 18:27:53 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644073.20990 Jan 3 18:28:08 service103 kernel: Lustre: 23102:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: 6958b9c2-373f-7b5f-0a19-4df6765beae3 reconnecting Jan 3 18:28:08 service103 kernel: Lustre: 23102:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from 6958b9c2-373f-7b5f-0a19-4df6765beae3@10.151.1.44@o2ib to 0xffff810bd2977a00; still busy with 1 active RPCs Jan 3 18:28:08 service103 kernel: Lustre: 23120:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0062: 1584acec-d396-3e73-847a-afc514e3b481 reconnecting Jan 3 18:28:08 service103 kernel: Lustre: 23120:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0062: refuse reconnection from 1584acec-d396-3e73-847a-afc514e3b481@10.151.54.69@o2ib to 0xffff810bd311bc00; still busy with 1 active RPCs Jan 3 18:28:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644102.20995 Jan 3 18:28:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644102.20950 Jan 3 18:28:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644102.21322 Jan 3 18:28:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644102.20998 Jan 3 18:28:22 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644102.21323 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.20996 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21325 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.20997 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21328 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21327 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21326 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21333 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21329 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21334 Jan 3 18:28:23 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644103.21330 Jan 3 18:28:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644104.21335 Jan 3 18:28:24 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644104.21336 Jan 3 18:28:25 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644105.21332 Jan 3 18:28:29 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644109.21339 Jan 3 18:28:32 service103 kernel: Lustre: 23156:0:(ldlm_lib.c:574:target_handle_reconnect()) nbp6-OST0002: 4e0761ad-b017-d616-966f-1ae431de83f7 reconnecting Jan 3 18:28:32 service103 kernel: Lustre: 23157:0:(ldlm_lib.c:874:target_handle_connect()) nbp6-OST0002: refuse reconnection from 76875829-9de1-df51-66cf-5d4ec644bcf3@10.151.21.159@o2ib to 0xffff810bc1521400; still busy with 1 active RPCs Jan 3 18:28:32 service103 kernel: Lustre: 23157:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message Jan 3 18:28:32 service103 kernel: Lustre: 23156:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8 previous similar messages Jan 3 18:28:37 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644117.21337 Jan 3 18:28:43 service103 kernel: Lustre: Failing over nbp6-OST0052 Jan 3 18:28:43 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST0052_UUID' is not available for connect (stopping) Jan 3 18:28:44 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST0052_UUID' is not available for connect (stopping) Jan 3 18:28:44 service103 kernel: LustreError: Skipped 25 previous similar messages Jan 3 18:28:45 service103 kernel: Lustre: Failing over nbp6-OST0042 Jan 3 18:28:45 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST0052_UUID' is not available for connect (stopping) Jan 3 18:28:45 service103 kernel: LustreError: Skipped 543 previous similar messages Jan 3 18:28:45 service103 kernel: Lustre: Failing over nbp6-OST001a Jan 3 18:28:45 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 18:28:45 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST006a_UUID' is not available for connect (stopping) Jan 3 18:28:45 service103 kernel: LustreError: Skipped 1200 previous similar messages Jan 3 18:28:45 service103 kernel: Lustre: Failing over nbp6-OST005a Jan 3 18:28:46 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST002a_UUID' is not available for connect (stopping) Jan 3 18:28:46 service103 kernel: LustreError: Skipped 4317 previous similar messages Jan 3 18:28:48 service103 kernel: Lustre: nbp6-OST006a: shutting down for failover; client state will be preserved. Jan 3 18:28:48 service103 kernel: Lustre: nbp6-OST0052: shutting down for failover; client state will be preserved. Jan 3 18:28:50 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST006a_UUID' is not available for connect (stopping) Jan 3 18:28:50 service103 kernel: LustreError: Skipped 18599 previous similar messages Jan 3 18:28:58 service103 kernel: Lustre: Service thread pid 21398 was inactive for 506.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 3 18:28:58 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:28:58 service103 kernel: Pid: 21398, comm: ll_ost_224 Jan 3 18:28:58 service103 kernel: Jan 3 18:28:58 service103 kernel: Call Trace: Jan 3 18:28:58 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:28:58 service103 kernel: [] list_add+0xc/0xe Jan 3 18:28:58 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:28:58 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:28:58 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:28:58 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:28:58 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:28:58 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:28:59 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:28:59 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:29:00 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:29:00 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:29:00 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:29:00 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:29:00 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:29:00 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:29:00 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:29:01 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:29:01 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:29:01 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:29:01 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:29:01 service103 kernel: Jan 3 18:29:01 service103 kernel: Pid: 21366, comm: ll_ost_222 Jan 3 18:29:01 service103 kernel: Jan 3 18:29:02 service103 kernel: Call Trace: Jan 3 18:29:02 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:29:02 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644138.21398 Jan 3 18:29:02 service103 kernel: [] list_add+0xc/0xe Jan 3 18:29:02 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:29:02 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:29:02 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:29:02 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:29:02 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:29:03 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:29:03 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:29:03 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:29:03 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:29:03 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST006a_UUID' is not available for connect (stopping) Jan 3 18:29:03 service103 kernel: LustreError: Skipped 34177 previous similar messages Jan 3 18:29:03 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:29:03 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:29:03 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:29:03 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:29:03 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:29:03 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:29:03 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:29:04 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:29:04 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:29:04 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:29:04 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:29:04 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:29:04 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:29:04 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:29:04 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:29:04 service103 kernel: Jan 3 18:29:04 service103 kernel: Pid: 21331, comm: ll_ost_211 Jan 3 18:29:04 service103 kernel: Jan 3 18:29:04 service103 kernel: Call Trace: Jan 3 18:29:04 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:29:04 service103 kernel: [] list_add+0xc/0xe Jan 3 18:29:05 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:29:05 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:29:05 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:29:05 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:29:05 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:29:05 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:29:05 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:29:05 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:29:05 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:29:05 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:29:05 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:29:05 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:29:05 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:29:06 service103 kernel: [] dequeue_task+0x18/0x37 Jan 3 18:29:06 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:29:06 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:29:06 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:29:06 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:29:06 service103 kernel: Jan 3 18:29:06 service103 kernel: Pid: 21349, comm: ll_ost_221 Jan 3 18:29:07 service103 kernel: Jan 3 18:29:07 service103 kernel: Call Trace: Jan 3 18:29:07 service103 kernel: [] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jan 3 18:29:07 service103 kernel: [] start_this_handle+0x301/0x3cb [jbd2] Jan 3 18:29:07 service103 kernel: [] autoremove_wake_function+0x0/0x2e Jan 3 18:29:07 service103 kernel: [] jbd2_journal_start+0xab/0xdf [jbd2] Jan 3 18:29:07 service103 kernel: [] ldiskfs_journal_start_sb+0x55/0xa0 [ldiskfs] Jan 3 18:29:07 service103 kernel: [] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs] Jan 3 18:29:07 service103 kernel: [] mntput_no_expire+0x19/0x88 Jan 3 18:29:07 service103 kernel: [] push_ctxt+0x370/0x380 [lvfs] Jan 3 18:29:07 service103 kernel: [] filter_client_add+0x508/0xc30 [obdfilter] Jan 3 18:29:07 service103 kernel: [] filter_export_stats_init+0x117/0x650 [obdfilter] Jan 3 18:29:07 service103 kernel: [] filter_connect+0x535/0x8c0 [obdfilter] Jan 3 18:29:08 service103 kernel: [] lustre_msg_add_op_flags+0x47/0x120 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] ost_handle+0x0/0x55b0 [ost] Jan 3 18:29:08 service103 kernel: [] target_handle_connect+0x21c6/0x2e80 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] ost_handle+0x8af/0x55b0 [ost] Jan 3 18:29:08 service103 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] __wake_up_common+0x3e/0x68 Jan 3 18:29:08 service103 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jan 3 18:29:08 service103 kernel: [] child_rip+0xa/0x11 Jan 3 18:29:08 service103 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Jan 3 18:29:09 service103 kernel: [] child_rip+0x0/0x11 Jan 3 18:29:09 service103 kernel: Jan 3 18:29:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644139.21349 Jan 3 18:29:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644140.21331 Jan 3 18:29:09 service103 kernel: LustreError: dumping log to /tmp/lustre-log.1325644140.21366 Jan 3 18:29:11 service103 kernel: Lustre: 23345:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply Jan 3 18:29:11 service103 kernel: req@ffff8109d64fe000 x1389011668757418/t0 o19->nbp6-mdtlov_UUID@MGC10.151.25.163@o2ib_0:0/0 lens 304/304 e 4 to 0 dl 1325644155 ref 2 fl Interpret:/0/0 rc 0/0 Jan 3 18:29:11 service103 kernel: Lustre: 23345:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Jan 3 18:29:17 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST0012_UUID' is not available for connect (stopping) Jan 3 18:29:17 service103 kernel: LustreError: Skipped 58797 previous similar messages Jan 3 18:29:55 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST000a_UUID' is not available for connect (stopping) Jan 3 18:29:55 service103 kernel: LustreError: 137-5: UUID 'nbp6-OST001a_UUID' is not available for connect (stopping) Jan 3 18:29:55 service103 kernel: LustreError: Skipped 61 previous similar messages Jan 3 18:29:55 service103 kernel: LustreError: Skipped 7 previous similar messages Jan 3 18:31:20 service103 kernel: Lustre: 19033:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-497), not sending early reply Jan 3 18:31:20 service103 kernel: req@ffff810aa5a16800 x1387753696563569/t0 o6->ad6b695a-6fca-bf3b-d50b-26454235af95@NET_0x500000a971a10_UUID:0/0 lens 512/400 e 1 to 0 dl 1325644285 ref 2 fl Interpret:/2/0 rc 0/0 Jan 3 18:31:20 service103 kernel: Lustre: 19033:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 56 previous similar messages Jan 3 18:31:23 service103 shutdown[23910]: shutting down for system reboot Jan 3 18:31:36 service103 kernel: SysRq : Terminate All Tasks Jan 3 18:31:36 service103 exiting on signal 15 Jan 3 18:34:45 service103 syslogd 1.4.1: restart. Jan 3 18:34:45 service103 kernel: klogd 1.4.1, log source = /proc/kmsg started. Jan 3 18:34:45 service103 kernel: Linux version 2.6.18-238.12.1.el5.20110722lustre186 (nobody@alcatraz) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Sun Jul 24 04:13:02 UTC 2011 Jan 3 18:34:45 service103 kernel: Command line: ro root=LABEL=sgiroot2 selinux=0 console=ttyS1,38400n8 crashkernel=128M@16M Jan 3 18:34:45 service103 kernel: BIOS-provided physical RAM map: Jan 3 18:34:45 service103 kernel: BIOS-e820: 0000000000010000 - 000000000009bc00 (usable) Jan 3 18:34:45 service103 kernel: BIOS-e820: 000000000009bc00 - 00000000000a0000 (reserved) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) Jan 3 18:34:45 service103 kernel: BIOS-e820: 0000000000100000 - 00000000bfef0000 (usable) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000bfef0000 - 00000000bff03000 (ACPI data) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000bff03000 - 00000000bff04000 (ACPI NVS) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000bff04000 - 00000000c0000000 (reserved) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Jan 3 18:34:45 service103 kernel: BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) Jan 3 18:34:46 service103 portmap[4238]: user rpc not found, reverting to user bin Jan 3 18:34:46 service103 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) Jan 3 18:34:46 service103 rpc.statd[4295]: Version 1.0.9 Starting Jan 3 18:34:46 service103 kernel: BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved) Jan 3 18:34:46 service103 kernel: BIOS-e820: 0000000100000000 - 0000000c40000000 (usable) Jan 3 18:34:46 service103 kernel: DMI present. Jan 3 18:34:46 service103 run_srp_daemon[4370]: failed srp_daemon: [HCA=mlx4_0] [port=1] [exit status=1]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 18:34:46 service103 run_srp_daemon[4371]: failed srp_daemon: [HCA=mlx4_1] [port=1] [exit status=1]. Will try to restart srp_daemon periodically. No more warnings will be issued in the next 7200 seconds if the same problem repeats Jan 3 18:34:48 service103 run_srp_daemon[4663]: starting srp_daemon: [HCA=mlx4_1] [port=1] Jan 3 18:34:48 service103 run_srp_daemon[4666]: starting srp_daemon: [HCA=mlx4_0] [port=1] Jan 3 18:34:49 service103 kernel: No NUMA configuration found Jan 3 18:34:49 service103 kernel: Faking a node at 0000000000000000-0000000c40000000 Jan 3 18:34:49 service103 kernel: Bootmem setup node 0 0000000000000000-0000000c40000000 Jan 3 18:34:50 service103 kernel: ACPI: PM-Timer IO Port: 0x1008 Jan 3 18:34:50 service103 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jan 3 18:34:50 service103 kernel: Processor #0 7:7 APIC version 20 Jan 3 18:34:50 service103 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x04] enabled) Jan 3 18:34:50 service103 kernel: Processor #4 7:7 APIC version 20 Jan 3 18:34:50 service103 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) Jan 3 18:34:50 service103 kernel: Processor #1 7:7 APIC version 20 Jan 3 18:34:50 service103 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x05] enabled) Jan 3 18:34:50 service103 kernel: Processor #5 7:7 APIC version 20 Jan 3 18:34:51 service103 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled) Jan 3 18:34:51 service103 kernel: Processor #2 7:7 APIC version 20 Jan 3 18:34:51 service103 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x06] enabled) Jan 3 18:34:51 service103 kernel: Processor #6 7:7 APIC version 20 Jan 3 18:34:51 service103 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled) Jan 3 18:34:51 service103 kernel: Processor #3 7:7 APIC version 20 Jan 3 18:34:51 service103 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) Jan 3 18:34:51 service103 kernel: Processor #7 7:7 APIC version 20 Jan 3 18:34:51 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1]) Jan 3 18:34:52 service103 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Jan 3 18:34:53 service103 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 Jan 3 18:34:53 service103 kernel: ACPI: IOAPIC (id[0x09] address[0xfec86000] gsi_base[24]) Jan 3 18:34:53 service103 kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec86000, GSI 24-47 Jan 3 18:34:53 service103 kernel: ACPI: IOAPIC (id[0x0a] address[0xfec89000] gsi_base[48]) Jan 3 18:34:53 service103 kernel: IOAPIC[2]: apic_id 10, version 32, address 0xfec89000, GSI 48-71 Jan 3 18:34:53 service103 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) Jan 3 18:34:54 service103 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jan 3 18:34:54 service103 kernel: Setting APIC routing to physical flat Jan 3 18:34:54 service103 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Jan 3 18:34:54 service103 kernel: Using ACPI (MADT) for SMP configuration information Jan 3 18:34:54 service103 kernel: Nosave address range: 000000000009b000 - 000000000009c000 Jan 3 18:34:54 service103 kernel: Nosave address range: 000000000009c000 - 00000000000a0000 Jan 3 18:34:54 service103 kernel: Nosave address range: 00000000000a0000 - 00000000000e0000 Jan 3 18:34:54 service103 kernel: Nosave address range: 00000000000e0000 - 0000000000100000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000bfef0000 - 00000000bff03000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000bff03000 - 00000000bff04000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000bff04000 - 00000000c0000000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000f0000000 - 00000000fec00000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000fec00000 - 00000000fec10000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000fec10000 - 00000000fee00000 Jan 3 18:34:55 service103 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000 Jan 3 18:34:56 service103 kernel: Nosave address range: 00000000fee01000 - 00000000ff000000 Jan 3 18:34:56 service103 kernel: Nosave address range: 00000000ff000000 - 0000000100000000 Jan 3 18:34:56 service103 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) Jan 3 18:34:56 service103 kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs Jan 3 18:34:56 service103 kernel: Built 1 zonelists. Total pages: 12405432 Jan 3 18:34:56 service103 kernel: Kernel command line: ro root=LABEL=sgiroot2 selinux=0 console=ttyS1,38400n8 crashkernel=128M@16M Jan 3 18:34:56 service103 kernel: Initializing CPU#0 Jan 3 18:34:56 service103 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Jan 3 18:34:56 service103 kernel: Console: colour VGA+ 80x25 Jan 3 18:34:57 service103 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Jan 3 18:34:57 service103 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Jan 3 18:34:57 service103 kernel: Checking aperture... Jan 3 18:34:57 service103 kernel: ACPI: DMAR not present Jan 3 18:34:57 service103 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Jan 3 18:34:57 service103 kernel: Placing software IO TLB between 0xf05e000 - 0x1305e000 Jan 3 18:34:57 service103 kernel: Memory: 49322844k/51380224k available (2665k kernel code, 1007248k reserved, 1746k data, 228k init) Jan 3 18:34:58 service103 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 5985.00 BogoMIPS (lpj=2992503) Jan 3 18:34:58 service103 kernel: kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved Jan 3 18:34:58 service103 kernel: Security Framework v1.0.0 initialized Jan 3 18:34:58 service103 kernel: SELinux: Disabled at boot. Jan 3 18:34:58 service103 kernel: Capability LSM initialized Jan 3 18:34:58 service103 kernel: Mount-cache hash table entries: 256 Jan 3 18:34:59 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:34:59 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:34:59 service103 kernel: using mwait in idle threads. Jan 3 18:34:59 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 18:34:59 service103 kernel: CPU: Processor Core ID: 0 Jan 3 18:34:59 service103 kernel: CPU0: Thermal monitoring enabled (TM2) Jan 3 18:35:00 service103 kernel: SMP alternatives: switching to UP code Jan 3 18:35:00 service103 kernel: ACPI: Core revision 20060707 Jan 3 18:35:00 service103 kernel: Using local APIC timer interrupts. Jan 3 18:35:00 service103 kernel: Detected 24.937 MHz APIC timer. Jan 3 18:35:00 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:01 service103 kernel: Booting processor 1/8 APIC 0x4 Jan 3 18:35:01 service103 kernel: Initializing CPU#1 Jan 3 18:35:01 service103 kernel: Calibrating delay using timer specific routine.. 5985.04 BogoMIPS (lpj=2992524) Jan 3 18:35:01 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:01 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:02 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 18:35:02 service103 kernel: CPU: Processor Core ID: 0 Jan 3 18:35:02 service103 kernel: CPU1: Thermal monitoring enabled (TM2) Jan 3 18:35:02 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:02 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:03 service103 kernel: Booting processor 2/8 APIC 0x1 Jan 3 18:35:03 service103 kernel: Initializing CPU#2 Jan 3 18:35:03 service103 kernel: Calibrating delay using timer specific routine.. 5985.01 BogoMIPS (lpj=2992506) Jan 3 18:35:03 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:03 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:03 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 18:35:03 service103 kernel: CPU: Processor Core ID: 1 Jan 3 18:35:04 service103 kernel: CPU2: Thermal monitoring enabled (TM2) Jan 3 18:35:04 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:04 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:04 service103 kernel: Booting processor 3/8 APIC 0x5 Jan 3 18:35:04 service103 kernel: Initializing CPU#3 Jan 3 18:35:04 service103 kernel: Calibrating delay using timer specific routine.. 5985.01 BogoMIPS (lpj=2992509) Jan 3 18:35:04 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:04 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:05 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 18:35:05 service103 kernel: CPU: Processor Core ID: 1 Jan 3 18:35:05 service103 kernel: CPU3: Thermal monitoring enabled (TM2) Jan 3 18:35:05 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:05 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:05 service103 kernel: Booting processor 4/8 APIC 0x2 Jan 3 18:35:05 service103 kernel: Initializing CPU#4 Jan 3 18:35:06 service103 kernel: Calibrating delay using timer specific routine.. 5984.99 BogoMIPS (lpj=2992497) Jan 3 18:35:06 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:06 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:06 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 18:35:06 service103 kernel: CPU: Processor Core ID: 2 Jan 3 18:35:06 service103 kernel: CPU4: Thermal monitoring enabled (TM2) Jan 3 18:35:06 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:07 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:07 service103 kernel: Booting processor 5/8 APIC 0x6 Jan 3 18:35:07 service103 kernel: Initializing CPU#5 Jan 3 18:35:07 service103 kernel: Calibrating delay using timer specific routine.. 5985.00 BogoMIPS (lpj=2992501) Jan 3 18:35:07 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:07 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:07 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 18:35:07 service103 kernel: CPU: Processor Core ID: 2 Jan 3 18:35:08 service103 kernel: CPU5: Thermal monitoring enabled (TM2) Jan 3 18:35:08 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:08 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:08 service103 kernel: Booting processor 6/8 APIC 0x3 Jan 3 18:35:08 service103 kernel: Initializing CPU#6 Jan 3 18:35:08 service103 kernel: Calibrating delay using timer specific routine.. 5984.99 BogoMIPS (lpj=2992498) Jan 3 18:35:08 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:08 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:09 service103 kernel: CPU: Physical Processor ID: 0 Jan 3 18:35:09 service103 kernel: CPU: Processor Core ID: 3 Jan 3 18:35:09 service103 kernel: CPU6: Thermal monitoring enabled (TM2) Jan 3 18:35:09 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:09 service103 kernel: SMP alternatives: switching to SMP code Jan 3 18:35:09 service103 kernel: Booting processor 7/8 APIC 0x7 Jan 3 18:35:09 service103 kernel: Initializing CPU#7 Jan 3 18:35:09 service103 kernel: Calibrating delay using timer specific routine.. 5985.00 BogoMIPS (lpj=2992502) Jan 3 18:35:10 service103 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jan 3 18:35:10 service103 kernel: CPU: L2 cache: 6144K Jan 3 18:35:10 service103 kernel: CPU: Physical Processor ID: 1 Jan 3 18:35:10 service103 kernel: CPU: Processor Core ID: 3 Jan 3 18:35:10 service103 kernel: CPU7: Thermal monitoring enabled (TM2) Jan 3 18:35:10 service103 kernel: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz stepping 06 Jan 3 18:35:10 service103 kernel: Brought up 8 CPUs Jan 3 18:35:12 service103 kernel: NMI watchdog testing PASSED. Jan 3 18:35:12 service103 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer. Jan 3 18:35:12 service103 kernel: time.c: Detected 2992.503 MHz processor. Jan 3 18:35:12 service103 kernel: migration_cost=8,9255 Jan 3 18:35:13 service103 kernel: checking if image is initramfs... it is Jan 3 18:35:13 service103 kernel: Freeing initrd memory: 2876k freed Jan 3 18:35:13 service103 kernel: NET: Registered protocol family 16 Jan 3 18:35:13 service103 kernel: ACPI: bus type pci registered Jan 3 18:35:13 service103 kernel: Warning: pci_mmcfg_init marking 256MB space uncacheable. Jan 3 18:35:13 service103 kernel: MCFG table requires 11MB uncacheable only. Try booting with acpi_mcfg_max_pci_bus_num=on Jan 3 18:35:13 service103 kernel: PCI: Using MMCONFIG at e0000000 Jan 3 18:35:13 service103 kernel: ACPI: Interpreter enabled Jan 3 18:35:13 service103 kernel: ACPI: Using IOAPIC for interrupt routing Jan 3 18:35:13 service103 kernel: ACPI: No dock devices found. Jan 3 18:35:13 service103 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Jan 3 18:35:13 service103 kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1 Jan 3 18:35:14 service103 kernel: PCI: Transparent bridge - 0000:00:1e.0 Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *10 11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 *11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 7 10 11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 *11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *7 10 11 14 15) Jan 3 18:35:15 service103 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 6 7 10 11 14 15) *9 Jan 3 18:35:16 service103 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Jan 3 18:35:16 service103 kernel: pnp: PnP ACPI init Jan 3 18:35:16 service103 kernel: pnp: PnP ACPI: found 12 devices Jan 3 18:35:16 service103 kernel: usbcore: registered new driver usbfs Jan 3 18:35:16 service103 kernel: usbcore: registered new driver hub Jan 3 18:35:16 service103 kernel: PCI: Using ACPI for IRQ routing Jan 3 18:35:16 service103 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report Jan 3 18:35:16 service103 kernel: NetLabel: Initializing Jan 3 18:35:16 service103 kernel: NetLabel: domain hash size = 128 Jan 3 18:35:16 service103 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jan 3 18:35:17 service103 kernel: NetLabel: unlabeled traffic allowed by default Jan 3 18:35:17 service103 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0 Jan 3 18:35:17 service103 kernel: hpet0: 3 64-bit timers, 14318180 Hz Jan 3 18:35:17 service103 kernel: ACPI: DMAR not present Jan 3 18:35:17 service103 kernel: PCI-GART: No AMD northbridge found. Jan 3 18:35:17 service103 kernel: pnp: 00:01: iomem range 0xe0000000-0xefffffff could not be reserved Jan 3 18:35:17 service103 kernel: pnp: 00:01: iomem range 0xfee00000-0xfee0ffff could not be reserved Jan 3 18:35:18 service103 kernel: pnp: 00:01: iomem range 0xfec86000-0xfec86fff has been reserved Jan 3 18:35:18 service103 kernel: pnp: 00:01: iomem range 0xfec89000-0xfec89fff has been reserved Jan 3 18:35:18 service103 kernel: PCI: Bridge: 0000:00:01.0 Jan 3 18:35:18 service103 kernel: IO window: disabled. Jan 3 18:35:18 service103 kernel: MEM window: d9200000-d92fffff Jan 3 18:35:18 service103 kernel: PREFETCH window 0x00000000d8000000-0x00000000d87fffff Jan 3 18:35:18 service103 kernel: PCI: Bridge: 0000:00:03.0 Jan 3 18:35:18 service103 kernel: IO window: 2000-2fff Jan 3 18:35:19 service103 kernel: MEM window: d9300000-d93fffff Jan 3 18:35:19 service103 kernel: PREFETCH window 0x00000000c2000000-0x00000000c21fffff Jan 3 18:35:19 service103 kernel: PCI: Bridge: 0000:00:05.0 Jan 3 18:35:19 service103 kernel: IO window: disabled. Jan 3 18:35:19 service103 kernel: MEM window: d9400000-d94fffff Jan 3 18:35:19 service103 kernel: PREFETCH window 0x00000000d8800000-0x00000000d8ffffff Jan 3 18:35:19 service103 kernel: PCI: Bridge: 0000:05:00.0 Jan 3 18:35:20 service103 kernel: IO window: disabled. Jan 3 18:35:20 service103 kernel: MEM window: disabled. Jan 3 18:35:20 service103 kernel: PREFETCH window: disabled. Jan 3 18:35:20 service103 kernel: PCI: Bridge: 0000:04:00.0 Jan 3 18:35:20 service103 kernel: IO window: disabled. Jan 3 18:35:20 service103 kernel: MEM window: disabled. Jan 3 18:35:20 service103 kernel: PREFETCH window: disabled. Jan 3 18:35:20 service103 kernel: PCI: Bridge: 0000:04:00.3 Jan 3 18:35:21 service103 kernel: IO window: disabled. Jan 3 18:35:21 service103 kernel: MEM window: disabled. Jan 3 18:35:21 service103 kernel: PREFETCH window: disabled. Jan 3 18:35:21 service103 kernel: PCI: Bridge: 0000:00:07.0 Jan 3 18:35:21 service103 kernel: IO window: disabled. Jan 3 18:35:21 service103 kernel: MEM window: d9500000-d95fffff Jan 3 18:35:21 service103 kernel: PREFETCH window: disabled. Jan 3 18:35:22 service103 kernel: PCI: Bridge: 0000:00:09.0 Jan 3 18:35:22 service103 kernel: IO window: 3000-3fff Jan 3 18:35:22 service103 kernel: MEM window: d9600000-d96fffff Jan 3 18:35:22 service103 kernel: PREFETCH window 0x00000000c2200000-0x00000000c22fffff Jan 3 18:35:22 service103 kernel: PCI: Bridge: 0000:00:1c.0 Jan 3 18:35:22 service103 kernel: IO window: disabled. Jan 3 18:35:22 service103 kernel: MEM window: disabled. Jan 3 18:35:23 service103 kernel: PREFETCH window: disabled. Jan 3 18:35:23 service103 kernel: PCI: Bridge: 0000:00:1e.0 Jan 3 18:35:23 service103 kernel: IO window: 4000-4fff Jan 3 18:35:23 service103 kernel: MEM window: d9700000-d97fffff Jan 3 18:35:23 service103 kernel: PREFETCH window 0x00000000d0000000-0x00000000d7ffffff Jan 3 18:35:23 service103 kernel: GSI 16 sharing vector 0xA9 and IRQ 16 Jan 3 18:35:24 service103 kernel: ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 48 (level, low) -> IRQ 169 Jan 3 18:35:24 service103 kernel: GSI 17 sharing vector 0xB1 and IRQ 17 Jan 3 18:35:25 service103 kernel: ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 50 (level, low) -> IRQ 177 Jan 3 18:35:25 service103 kernel: GSI 18 sharing vector 0xB9 and IRQ 18 Jan 3 18:35:25 service103 kernel: ACPI: PCI Interrupt 0000:00:05.0[A] -> GSI 52 (level, low) -> IRQ 185 Jan 3 18:35:25 service103 kernel: GSI 19 sharing vector 0xC1 and IRQ 19 Jan 3 18:35:26 service103 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 18:35:26 service103 kernel: ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 18:35:26 service103 kernel: ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 54 (level, low) -> IRQ 193 Jan 3 18:35:27 service103 kernel: GSI 20 sharing vector 0xC9 and IRQ 20 Jan 3 18:35:27 service103 kernel: ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 56 (level, low) -> IRQ 201 Jan 3 18:35:27 service103 kernel: GSI 21 sharing vector 0xD1 and IRQ 21 Jan 3 18:35:27 service103 kernel: ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 209 Jan 3 18:35:27 service103 kernel: NET: Registered protocol family 2 Jan 3 18:35:28 service103 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Jan 3 18:35:28 service103 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Jan 3 18:35:28 service103 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jan 3 18:35:28 service103 kernel: TCP: Hash tables configured (established 262144 bind 65536) Jan 3 18:35:28 service103 kernel: TCP reno registered Jan 3 18:35:28 service103 kernel: Simple Boot Flag at 0x41 set to 0x80 Jan 3 18:35:28 service103 kernel: audit: initializing netlink socket (disabled) Jan 3 18:35:28 service103 kernel: type=2000 audit(1325615618.625:1): initialized Jan 3 18:35:28 service103 kernel: Total HugeTLB memory allocated, 0 Jan 3 18:35:29 service103 kernel: VFS: Disk quotas dquot_6.5.1 Jan 3 18:35:29 service103 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jan 3 18:35:29 service103 kernel: Initializing Cryptographic API Jan 3 18:35:29 service103 kernel: alg: No test for crc32c (crc32c-generic) Jan 3 18:35:29 service103 kernel: ksign: Installing public key data Jan 3 18:35:29 service103 kernel: Loading keyring Jan 3 18:35:29 service103 kernel: io scheduler noop registered Jan 3 18:35:29 service103 kernel: io scheduler anticipatory registered Jan 3 18:35:29 service103 kernel: io scheduler deadline registered Jan 3 18:35:30 service103 kernel: io scheduler cfq registered (default) Jan 3 18:35:31 service103 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jan 3 18:35:31 service103 kernel: ACPI: Processor [CPU0] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU1] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU2] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU3] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU4] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU5] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU6] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: ACPI: Processor [CPU7] (supports 8 throttling states) Jan 3 18:35:32 service103 kernel: Real Time Clock Driver v1.12ac Jan 3 18:35:32 service103 kernel: Non-volatile memory driver v1.2 Jan 3 18:35:32 service103 kernel: Linux agpgart interface v0.101 (c) Dave Jones Jan 3 18:35:32 service103 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Jan 3 18:35:33 service103 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 3 18:35:33 service103 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jan 3 18:35:33 service103 kernel: 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 3 18:35:33 service103 kernel: 00:0b: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jan 3 18:35:33 service103 kernel: brd: module loaded Jan 3 18:35:33 service103 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Jan 3 18:35:33 service103 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Jan 3 18:35:33 service103 kernel: ESB2: IDE controller at PCI slot 0000:00:1f.1 Jan 3 18:35:34 service103 kernel: ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 16 (level, low) -> IRQ 209 Jan 3 18:35:34 service103 kernel: ESB2: chipset revision 9 Jan 3 18:35:34 service103 kernel: ESB2: not 100% native mode: will probe irqs later Jan 3 18:35:34 service103 kernel: ide0: BM-DMA at 0x1860-0x1867, BIOS settings: hda:pio, hdb:DMA Jan 3 18:35:34 service103 kernel: hdb: MATSHITADVD-RAM UJ870PC, ATAPI CD/DVD-ROM drive Jan 3 18:35:34 service103 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 Jan 3 18:35:35 service103 kernel: ide-floppy driver 0.99.newide Jan 3 18:35:35 service103 kernel: usbcore: registered new driver hiddev Jan 3 18:35:35 service103 kernel: usbcore: registered new driver usbhid Jan 3 18:35:35 service103 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Jan 3 18:35:35 service103 kernel: PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12 Jan 3 18:35:35 service103 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jan 3 18:35:35 service103 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jan 3 18:35:36 service103 kernel: mice: PS/2 mouse device common for all mice Jan 3 18:35:36 service103 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 Jan 3 18:35:36 service103 kernel: md: bitmap version 4.39 Jan 3 18:35:36 service103 kernel: TCP bic registered Jan 3 18:35:36 service103 kernel: Initializing IPsec netlink socket Jan 3 18:35:36 service103 kernel: NET: Registered protocol family 1 Jan 3 18:35:36 service103 kernel: NET: Registered protocol family 17 Jan 3 18:35:36 service103 kernel: ACPI: (supports S0 S1 S4 S5) Jan 3 18:35:37 service103 kernel: Initalizing network drop monitor service Jan 3 18:35:37 service103 kernel: Freeing unused kernel memory: 228k freed Jan 3 18:35:37 service103 kernel: Write protecting the kernel read-only data: 600k Jan 3 18:35:37 service103 kernel: SCSI subsystem initialized Jan 3 18:35:37 service103 kernel: GSI 22 sharing vector 0x5A and IRQ 22 Jan 3 18:35:38 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 90 Jan 3 18:35:38 service103 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller Jan 3 18:35:38 service103 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1 Jan 3 18:35:38 service103 kernel: ehci_hcd 0000:00:1d.7: debug port 1 Jan 3 18:35:38 service103 kernel: ehci_hcd 0000:00:1d.7: irq 90, io mem 0xd9804000 Jan 3 18:35:39 service103 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Jan 3 18:35:39 service103 kernel: usb usb1: configuration #1 chosen from 1 choice Jan 3 18:35:39 service103 kernel: hub 1-0:1.0: USB hub found Jan 3 18:35:39 service103 kernel: hub 1-0:1.0: 6 ports detected Jan 3 18:35:39 service103 kernel: USB Universal Host Controller Interface driver v3.0 Jan 3 18:35:39 service103 kernel: GSI 23 sharing vector 0x62 and IRQ 23 Jan 3 18:35:40 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 20 (level, low) -> IRQ 98 Jan 3 18:35:40 service103 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller Jan 3 18:35:40 service103 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 Jan 3 18:35:40 service103 kernel: uhci_hcd 0000:00:1d.0: irq 98, io base 0x00001800 Jan 3 18:35:40 service103 kernel: usb usb2: configuration #1 chosen from 1 choice Jan 3 18:35:40 service103 kernel: hub 2-0:1.0: USB hub found Jan 3 18:35:40 service103 kernel: hub 2-0:1.0: 2 ports detected Jan 3 18:35:41 service103 kernel: usb 1-6: new high speed USB device using ehci_hcd and address 2 Jan 3 18:35:41 service103 kernel: GSI 24 sharing vector 0x6A and IRQ 24 Jan 3 18:35:41 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 21 (level, low) -> IRQ 106 Jan 3 18:35:41 service103 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller Jan 3 18:35:41 service103 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3 Jan 3 18:35:41 service103 kernel: uhci_hcd 0000:00:1d.1: irq 106, io base 0x00001820 Jan 3 18:35:41 service103 kernel: usb usb3: configuration #1 chosen from 1 choice Jan 3 18:35:41 service103 kernel: hub 3-0:1.0: USB hub found Jan 3 18:35:41 service103 kernel: hub 3-0:1.0: 2 ports detected Jan 3 18:35:42 service103 kernel: usb 1-6: configuration #1 chosen from 1 choice Jan 3 18:35:42 service103 kernel: input: Peppercon AG Multidevice as /class/input/input0 Jan 3 18:35:42 service103 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1d.7-6 Jan 3 18:35:42 service103 kernel: input: Peppercon AG Multidevice as /class/input/input1 Jan 3 18:35:42 service103 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1d.7-6 Jan 3 18:35:42 service103 kernel: GSI 25 sharing vector 0x72 and IRQ 25 Jan 3 18:35:42 service103 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 22 (level, low) -> IRQ 114 Jan 3 18:35:43 service103 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller Jan 3 18:35:43 service103 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4 Jan 3 18:35:43 service103 kernel: uhci_hcd 0000:00:1d.2: irq 114, io base 0x00001840 Jan 3 18:35:43 service103 kernel: usb usb4: configuration #1 chosen from 1 choice Jan 3 18:35:43 service103 kernel: hub 4-0:1.0: USB hub found Jan 3 18:35:43 service103 kernel: hub 4-0:1.0: 2 ports detected Jan 3 18:35:44 service103 kernel: Fusion MPT base driver 3.04.15rh Jan 3 18:35:44 service103 kernel: Copyright (c) 1999-2008 LSI Corporation Jan 3 18:35:44 service103 kernel: Fusion MPT SAS Host driver 3.04.15rh Jan 3 18:35:44 service103 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 50 (level, low) -> IRQ 177 Jan 3 18:35:44 service103 kernel: mptbase: ioc0: Initiating bringup Jan 3 18:35:45 service103 kernel: ioc0: LSISAS1068E B3: Capabilities={Initiator} Jan 3 18:35:45 service103 kernel: scsi0 : ioc0: LSISAS1068E B3, FwRev=01170400h, Ports=1, MaxQ=286, IRQ=177 Jan 3 18:35:45 service103 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 5, phy 0, sas_addr 0x3f2e56397eaa8476 Jan 3 18:35:46 service103 kernel: Vendor: ATA Model: HDS725050KLA360 Rev: AD1A Jan 3 18:35:46 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:35:46 service103 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 2, phy 1, sas_addr 0x3f2e563b87908979 Jan 3 18:35:46 service103 kernel: Vendor: ATA Model: HDS725050KLA360 Rev: AD1A Jan 3 18:35:46 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:35:46 service103 kernel: mptsas: ioc0: attaching raid volume, channel 1, id 0 Jan 3 18:35:46 service103 kernel: Vendor: LSILOGIC Model: Logical Volume Rev: 3000 Jan 3 18:35:46 service103 kernel: Type: Direct-Access ANSI SCSI revision: 02 Jan 3 18:35:47 service103 kernel: SCSI device sda: 976482304 512-byte hdwr sectors (499959 MB) Jan 3 18:35:47 service103 kernel: sda: Write Protect is off Jan 3 18:35:47 service103 kernel: SCSI device sda: drive cache: write through Jan 3 18:35:47 service103 kernel: SCSI device sda: 976482304 512-byte hdwr sectors (499959 MB) Jan 3 18:35:47 service103 kernel: sda: Write Protect is off Jan 3 18:35:47 service103 kernel: SCSI device sda: drive cache: write through Jan 3 18:35:47 service103 kernel: sda: sda1 sda2 sda3 < sda5 sda6 sda7 sda8 > Jan 3 18:35:47 service103 kernel: sd 0:1:0:0: Attached scsi disk sda Jan 3 18:35:47 service103 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Jan 3 18:35:47 service103 kernel: device-mapper: uevent: version 1.0.3 Jan 3 18:35:47 service103 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com Jan 3 18:35:47 service103 kernel: device-mapper: dm-raid45: initialized v0.2594l Jan 3 18:35:48 service103 kernel: Fusion MPT FC Host driver 3.04.15rh Jan 3 18:35:48 service103 kernel: Fusion MPT misc device (ioctl) driver 3.04.15rh Jan 3 18:35:48 service103 kernel: mptctl: Registered with Fusion MPT base driver Jan 3 18:35:48 service103 kernel: mptctl: /dev/mptctl @ (major,minor=10,220) Jan 3 18:35:48 service103 kernel: BIOS EDD facility v0.16 2004-Jun-25, 1 devices found Jan 3 18:35:48 service103 kernel: megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006) Jan 3 18:35:48 service103 kernel: megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006) Jan 3 18:35:48 service103 kernel: megasas: 00.00.04.31-RH1 Tues. June. 15 14:13:02 EST 2010 Jan 3 18:35:48 service103 kernel: 802.1Q VLAN Support v1.8 Ben Greear Jan 3 18:35:48 service103 kernel: All bugs added by David S. Miller Jan 3 18:35:48 service103 kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Jan 3 18:35:48 service103 kernel: EXT3-fs: write access will be enabled during recovery. Jan 3 18:35:48 service103 kernel: kjournald starting. Commit interval 5 seconds Jan 3 18:35:48 service103 kernel: EXT3-fs: recovery complete. Jan 3 18:35:49 service103 kernel: EXT3-fs: mounted filesystem with ordered data mode. Jan 3 18:35:49 service103 kernel: dca service started, version 1.8 Jan 3 18:35:49 service103 kernel: intel_rng: FWH not detected Jan 3 18:35:49 service103 kernel: input: PC Speaker as /class/input/input2 Jan 3 18:35:49 service103 kernel: EDAC MC: Ver: 2.0.1 Jul 24 2011 Jan 3 18:35:49 service103 kernel: EDAC MC0: Giving out device to i5400_edac.c I5400: DEV 0000:00:10.0 Jan 3 18:35:49 service103 kernel: hdb: ATAPI 24X DVD-ROM DVD-R-RAM CD-R/RW drive, 2048kB Cache, UDMA(33) Jan 3 18:35:49 service103 kernel: Uniform CD-ROM driver Revision: 3.20 Jan 3 18:35:49 service103 kernel: memtrack::init_module done. Jan 3 18:35:49 service103 kernel: GSI 26 sharing vector 0x7A and IRQ 26 Jan 3 18:35:49 service103 kdump: kexec: loaded kdump kernel Jan 3 18:35:49 service103 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 122 Jan 3 18:35:50 service103 kdump: started up Jan 3 18:35:50 service103 kernel: Initializing USB Mass Storage driver... Jan 3 18:35:50 service103 kernel: scsi1 : SCSI emulation for USB Mass Storage devices Jan 3 18:35:50 service103 kernel: usbcore: registered new driver usb-storage Jan 3 18:35:50 service103 kernel: USB Mass Storage support registered. Jan 3 18:35:50 service103 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 0 Jan 3 18:35:51 service103 kernel: scsi 0:0:1:0: Attached scsi generic sg1 type 0 Jan 3 18:35:51 service103 kernel: sd 0:1:0:0: Attached scsi generic sg2 type 0 Jan 3 18:35:51 service103 hcid[5857]: Bluetooth HCI daemon Jan 3 18:35:51 service103 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2-1 Jan 3 18:35:51 service103 kernel: Copyright (c) 2007-2009 Intel Corporation. Jan 3 18:35:51 service103 hcid[5857]: Can't open system message bus connection: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory Jan 3 18:35:51 service103 sdpd[5861]: Bluetooth SDP daemon Jan 3 18:35:51 service103 kernel: ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 56 (level, low) -> IRQ 201 Jan 3 18:35:51 service103 hcid[5857]: Unable to get on D-Bus Jan 3 18:35:52 service103 kernel: igb 0000:08:00.0: Disabling ASPM L0s upstream switch port 0000:00:09.0 Jan 3 18:35:52 service103 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready. Jan 3 18:35:52 service103 kernel: igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection Jan 3 18:35:52 service103 hidd[5972]: Bluetooth HID daemon Jan 3 18:35:52 service103 kernel: igb 0000:08:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:30:48:c4:4f:0c Jan 3 18:35:53 service103 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead. Jan 3 18:35:53 service103 kernel: igb 0000:08:00.0: eth0: PBA No: ffffff-0ff Jan 3 18:35:53 service103 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s) Jan 3 18:35:53 service103 kernel: igb 0000:08:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Jan 3 18:35:53 service103 kernel: GSI 27 sharing vector 0xB2 and IRQ 27 Jan 3 18:35:53 service103 kernel: ACPI: PCI Interrupt 0000:08:00.1[B] -> GSI 70 (level, low) -> IRQ 178 Jan 3 18:35:53 service103 kernel: igb 0000:08:00.1: Disabling ASPM L0s upstream switch port 0000:00:09.0 Jan 3 18:35:54 service103 kernel: igb 0000:08:00.1: Intel(R) Gigabit Ethernet Network Connection Jan 3 18:35:54 service103 kernel: igb 0000:08:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:30:48:c4:4f:0d Jan 3 18:35:54 service103 kernel: igb 0000:08:00.1: eth1: PBA No: ffffff-0ff Jan 3 18:35:54 service103 kernel: igb 0000:08:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Jan 3 18:35:54 service103 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011) Jan 3 18:35:54 service103 kernel: mlx4_core: Initializing 0000:01:00.0 Jan 3 18:35:54 service103 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 48 (level, low) -> IRQ 169 Jan 3 18:35:54 service103 kernel: mlx4_core: Initializing 0000:03:00.0 Jan 3 18:35:54 service103 kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 52 (level, low) -> IRQ 185 Jan 3 18:35:54 service103 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Jan 3 18:35:54 service103 kernel: Type: CD-ROM ANSI SCSI revision: 03 Jan 3 18:35:54 service103 kernel: scsi 1:0:0:0: Attached scsi generic sg3 type 5 Jan 3 18:35:55 service103 kernel: sr0: scsi-1 drive Jan 3 18:35:55 service103 kernel: floppy0: no floppy controllers found Jan 3 18:35:55 service103 kernel: work still pending Jan 3 18:35:55 service103 kernel: lp: driver loaded but no devices found Jan 3 18:35:55 service103 kernel: ACPI: Power Button (FF) [PWRF] Jan 3 18:35:55 service103 kernel: ACPI: Power Button (CM) [PWRB] Jan 3 18:35:55 service103 kernel: ACPI: Mapper loaded Jan 3 18:35:55 service103 kernel: dell-wmi: No known WMI GUID found Jan 3 18:35:55 service103 kernel: md: Autodetecting RAID arrays. Jan 3 18:35:55 service103 kernel: md: autorun ... Jan 3 18:35:55 service103 kernel: md: ... autorun DONE. Jan 3 18:35:55 service103 kernel: device-mapper: multipath: version 1.0.6 loaded Jan 3 18:35:55 service103 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded Jan 3 18:35:55 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 18:35:56 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 18:35:56 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 18:35:56 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 18:35:56 service103 kernel: EXT3 FS on sda8, internal journal Jan 3 18:35:56 service103 kernel: kjournald starting. Commit interval 5 seconds Jan 3 18:35:56 service103 kernel: EXT3 FS on sda7, internal journal Jan 3 18:35:56 service103 kernel: EXT3-fs: mounted filesystem with ordered data mode. Jan 3 18:35:56 service103 kernel: Adding 2000052k swap on /dev/sda1. Priority:-1 extents:1 across:2000052k Jan 3 18:35:56 service103 kernel: IA-32 Microcode Update Driver: v1.14a Jan 3 18:35:56 service103 kernel: microcode: CPU1 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:56 service103 /etc/init.d/memlog[6070]: WARNING: Could not load module(s): worm Jan 3 18:35:56 service103 kernel: microcode: CPU2 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU3 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU7 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU5 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU6 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU4 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: microcode: CPU0 updated from revision 0x60b to 0x60f, date = 09292010 Jan 3 18:35:57 service103 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011) Jan 3 18:35:57 service103 kernel: NET: Registered protocol family 10 Jan 3 18:35:57 service103 kernel: lo: Disabled Privacy Extensions Jan 3 18:35:57 service103 kernel: IPv6 over IPv4 tunneling driver Jan 3 18:35:57 service103 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready Jan 3 18:35:57 service103 kernel: ib0: enabling connected mode will cause multicast packet drops Jan 3 18:35:58 service103 kernel: ib0: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:58 service103 kernel: ib0: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:58 service103 kernel: ADDRCONF(NETDEV_UP): ib1: link is not ready Jan 3 18:35:58 service103 automount[6097]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master Jan 3 18:35:58 service103 kernel: ib1: enabling connected mode will cause multicast packet drops Jan 3 18:35:58 service103 kernel: ib1: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:58 service103 nscd: 6115 Failed to run nscd as user 'nscd' Jan 3 18:35:58 service103 kernel: ib1: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:58 service103 kernel: ib2: enabling connected mode will cause multicast packet drops Jan 3 18:35:59 service103 kernel: ib2: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:59 service103 kernel: ib2: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:59 service103 kernel: ib3: enabling connected mode will cause multicast packet drops Jan 3 18:35:59 service103 kernel: ib3: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:59 service103 kernel: ib3: mtu > 2044 will cause multicast packet drops. Jan 3 18:35:59 service103 kernel: Loading iSCSI transport class v2.0-871. Jan 3 18:35:59 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client Jan 3 18:35:59 service103 kernel: cxgb3i: Unknown symbol cxgb3_register_client Jan 3 18:35:59 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid Jan 3 18:35:59 service103 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid Jan 3 18:36:00 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get Jan 3 18:36:00 service103 kernel: cxgb3i: Unknown symbol t3_l2t_get Jan 3 18:36:00 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid Jan 3 18:36:00 service103 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid Jan 3 18:36:00 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free Jan 3 18:36:00 service103 kernel: cxgb3i: Unknown symbol t3_l2e_free Jan 3 18:36:00 service103 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow Jan 3 18:36:00 service103 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow Jan 3 18:36:00 service103 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client Jan 3 18:36:00 service103 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client Jan 3 18:36:00 service103 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010) Jan 3 18:36:01 service103 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010) Jan 3 18:36:01 service103 kernel: iscsi: registered transport (bnx2i) Jan 3 18:36:01 service103 kernel: iscsi: registered transport (tcp) Jan 3 18:36:01 service103 kernel: iscsi: registered transport (be2iscsi) Jan 3 18:36:01 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 18:36:01 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 18:36:01 service103 kernel: device-mapper: table: 253:0: multipath: error getting device Jan 3 18:36:01 service103 hpiod: 1.6.7 accepting connections at 2208... Jan 3 18:36:01 service103 kernel: device-mapper: ioctl: error adding target to table Jan 3 18:36:02 service103 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jan 3 18:36:02 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jan 3 18:36:02 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_0 Jan 3 18:36:02 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Jan 3 18:36:02 service103 kernel: ADDRCONF(NETDEV_CHANGE): ib1: link becomes ready Jan 3 18:36:03 service103 kernel: Bluetooth: Core ver 2.10 Jan 3 18:36:03 service103 kernel: NET: Registered protocol family 31 Jan 3 18:36:03 service103 kernel: Bluetooth: HCI device and connection manager initialized Jan 3 18:36:03 service103 kernel: Bluetooth: HCI socket layer initialized Jan 3 18:36:03 service103 kernel: Bluetooth: L2CAP ver 2.8 Jan 3 18:36:03 service103 kernel: Bluetooth: L2CAP socket layer initialized Jan 3 18:36:03 service103 kernel: Bluetooth: RFCOMM socket layer initialized Jan 3 18:36:03 service103 kernel: Bluetooth: RFCOMM TTY layer initialized Jan 3 18:36:04 service103 kernel: Bluetooth: RFCOMM ver 1.8 Jan 3 18:36:04 service103 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 Jan 3 18:36:04 service103 kernel: ipmi message handler version 39.1 Jan 3 18:36:04 service103 kernel: IPMI System Interface driver. Jan 3 18:36:04 service103 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 Jan 3 18:36:04 service103 kernel: ipmi: Found new BMC (man_id: 0x0028c5, prod_id: 0x0004, dev_id: 0x22) Jan 3 18:36:04 service103 kernel: IPMI kcs interface initialized Jan 3 18:36:05 service103 kernel: ipmi device interface Jan 3 18:36:06 service103 OpenSM[6238]: Jan 3 18:36:06 service103 OpenSM[6238]: ONBOOT=no Jan 3 18:36:06 service103 OpenSM[6238]: Loading Cached Option:guid = 0x0002c903000f9f83 Jan 3 18:36:06 service103 OpenSM[6238]: Loading Cached Option:honor_guid2lid_file = TRUE Jan 3 18:36:06 service103 OpenSM[6238]: Loading Cached Option:log_file = /var/log/opensm-mlx4_0_1.log Jan 3 18:36:06 service103 OpenSM[6238]: Loading Cached Option:dump_files_dir = /var/cache/opensm/mlx4_0_1 Jan 3 18:36:06 service103 OpenSM[6240]: /var/log/opensm-mlx4_0_1.log log file opened Jan 3 18:36:06 service103 OpenSM[6240]: OpenSM 3.3.7 Jan 3 18:36:06 service103 OpenSM[6240]: Entering DISCOVERING state Jan 3 18:36:06 service103 OpenSM[6240]: Entering MASTER state Jan 3 18:36:06 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jan 3 18:36:06 service103 OpenSM[6240]: SUBNET UP Jan 3 18:36:06 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_0 Jan 3 18:36:06 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Jan 3 18:36:07 service103 kernel: scsi2 : SRP.T10:1A6D0F0003C90200 Jan 3 18:36:07 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:07 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 18:36:07 service103 kernel: scsi 2:0:0:0: Attached scsi generic sg4 type 12 Jan 3 18:36:07 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:07 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:07 service103 kernel: sd 2:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 18:36:07 service103 kernel: sdb: Unit Not Ready, sense: Jan 3 18:36:07 service103 kernel: : Current: sense key: Unit Attention Jan 3 18:36:07 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 18:36:07 service103 OpenSM[6273]: Jan 3 18:36:07 service103 kernel: Jan 3 18:36:07 service103 multipathd: sdb: add path (uevent) Jan 3 18:36:07 service103 OpenSM[6273]: ONBOOT=no Jan 3 18:36:07 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 18:36:07 service103 OpenSM[6273]: Loading Cached Option:guid = 0x0002c903000f9f8f Jan 3 18:36:07 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:07 service103 OpenSM[6273]: Loading Cached Option:honor_guid2lid_file = TRUE Jan 3 18:36:07 service103 kernel: sdb: Write Protect is off Jan 3 18:36:08 service103 OpenSM[6273]: Loading Cached Option:log_file = /var/log/opensm-mlx4_1_1.log Jan 3 18:36:08 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:16 10] Jan 3 18:36:08 service103 OpenSM[6273]: Loading Cached Option:dump_files_dir = /var/cache/opensm/mlx4_1_1 Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 kernel: SCSI device sdb: drive cache: write back w/ FUA Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 multipathd: ddn6a-nbp6-ost2: event checker started Jan 3 18:36:08 service103 OpenSM[6529]: /var/log/opensm-mlx4_1_1.log log file opened Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 logger: Adjusted sdb max_sectors_kb=4096 Jan 3 18:36:08 service103 logger: Adjusted sdc max_sectors_kb=4096 Jan 3 18:36:08 service103 kernel: sdb : very big device. try to use READ CAPACITY(16). Jan 3 18:36:08 service103 logger: Adjusted sdd max_sectors_kb=4096 Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 multipathd: sdc: add path (uevent) Jan 3 18:36:08 service103 logger: Adjusted blockdev Jan 3 18:36:08 service103 OpenSM[6529]: OpenSM 3.3.7 Jan 3 18:36:09 service103 logger: Adjusted sde max_sectors_kb=4096 Jan 3 18:36:09 service103 logger: Adjusted sdb scheduler=deadline Jan 3 18:36:09 service103 logger: Adjusted sdc scheduler=deadline Jan 3 18:36:09 service103 kernel: SCSI device sdb: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:09 service103 logger: Adjusted blockdev Jan 3 18:36:09 service103 logger: Adjusted sdd scheduler=deadline Jan 3 18:36:09 service103 logger: Adjusted sdf max_sectors_kb=4096 Jan 3 18:36:09 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:32 10] Jan 3 18:36:09 service103 logger: Adjusted sdg max_sectors_kb=4096 Jan 3 18:36:09 service103 OpenSM[6529]: Entering DISCOVERING state Jan 3 18:36:09 service103 logger: Adjusted sde scheduler=deadline Jan 3 18:36:09 service103 logger: Adjected sdb timeout=280 Jan 3 18:36:09 service103 logger: Adjected sdc timeout=280 Jan 3 18:36:09 service103 logger: Adjusted blockdev Jan 3 18:36:09 service103 kernel: sdb: Write Protect is off Jan 3 18:36:09 service103 logger: Adjusted sdh max_sectors_kb=4096 Jan 3 18:36:09 service103 logger: Adjected sdd timeout=280 Jan 3 18:36:09 service103 logger: Adjusted blockdev Jan 3 18:36:09 service103 logger: Adjusted sdf scheduler=deadline Jan 3 18:36:09 service103 multipathd: ddn6a-nbp6-ost10: event checker started Jan 3 18:36:09 service103 logger: Adjusted sdg scheduler=deadline Jan 3 18:36:09 service103 logger: Adjusted blockdev Jan 3 18:36:10 service103 OpenSM[6529]: Entering MASTER state Jan 3 18:36:10 service103 logger: Adjected sde timeout=280 Jan 3 18:36:10 service103 boot.booted: TEMPO:service103 EVENT:NODE_BOOTED APP:BOOT.BOOTED DATE:Jan 3 2012 18:36:10 VERSION:1.0 TEXT:Node booted successfully. Jan 3 18:36:10 service103 logger: Adjusted blockdev Jan 3 18:36:10 service103 logger: Adjusted sdj max_sectors_kb=4096 Jan 3 18:36:10 service103 logger: Adjusted blockdev Jan 3 18:36:10 service103 logger: Adjusted blockdev Jan 3 18:36:10 service103 logger: Adjusted sdh scheduler=deadline Jan 3 18:36:10 service103 logger: Adjusted sdk max_sectors_kb=4096 Jan 3 18:36:10 service103 logger: Adjected sdf timeout=280 Jan 3 18:36:10 service103 multipathd: sdd: add path (uevent) Jan 3 18:36:10 service103 logger: Adjected sdg timeout=280 Jan 3 18:36:10 service103 logger: Adjusted blockdev Jan 3 18:36:10 service103 logger: Adjusted sdl max_sectors_kb=4096 Jan 3 18:36:10 service103 OpenSM[6529]: SUBNET UP Jan 3 18:36:10 service103 logger: Adjusted sdm max_sectors_kb=4096 Jan 3 18:36:11 service103 logger: Adjusted sdj scheduler=deadline Jan 3 18:36:11 service103 logger: Adjusted sdn max_sectors_kb=4096 Jan 3 18:36:11 service103 kernel: SCSI device sdb: drive cache: write back w/ FUA Jan 3 18:36:11 service103 logger: Adjusted sdo max_sectors_kb=4096 Jan 3 18:36:11 service103 logger: Adjected sdh timeout=280 Jan 3 18:36:11 service103 logger: Adjusted sdk scheduler=deadline Jan 3 18:36:11 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:48 10] Jan 3 18:36:11 service103 logger: Adjusted sdp max_sectors_kb=4096 Jan 3 18:36:11 service103 logger: Adjusted sdl scheduler=deadline Jan 3 18:36:12 service103 logger: Adjusted sdm scheduler=deadline Jan 3 18:36:12 service103 logger: Adjected sdj timeout=280 Jan 3 18:36:12 service103 kernel: sdb: unknown partition table Jan 3 18:36:12 service103 logger: Adjusted sdn scheduler=deadline Jan 3 18:36:12 service103 logger: Adjusted sdo scheduler=deadline Jan 3 18:36:12 service103 logger: Adjected sdk timeout=280 Jan 3 18:36:12 service103 multipathd: ddn6a-nbp6-ost18: event checker started Jan 3 18:36:12 service103 logger: Adjusted sdp scheduler=deadline Jan 3 18:36:12 service103 logger: Adjected sdl timeout=280 Jan 3 18:36:12 service103 logger: Adjected sdm timeout=280 Jan 3 18:36:13 service103 kernel: sd 2:0:0:3: Attached scsi disk sdb Jan 3 18:36:13 service103 logger: Adjected sdn timeout=280 Jan 3 18:36:13 service103 logger: Adjected sdo timeout=280 Jan 3 18:36:13 service103 multipathd: sde: add path (uevent) Jan 3 18:36:13 service103 logger: Adjected sdp timeout=280 Jan 3 18:36:13 service103 kernel: sd 2:0:0:3: Attached scsi generic sg5 type 0 Jan 3 18:36:13 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:64 10] Jan 3 18:36:14 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:14 service103 multipathd: ddn6a-nbp6-ost26: event checker started Jan 3 18:36:14 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:14 service103 multipathd: sdf: add path (uevent) Jan 3 18:36:14 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 18:36:14 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:80 10] Jan 3 18:36:14 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:14 service103 multipathd: ddn6a-nbp6-ost34: event checker started Jan 3 18:36:14 service103 kernel: sdc: Write Protect is off Jan 3 18:36:14 service103 multipathd: dm-0: add map (uevent) Jan 3 18:36:14 service103 multipathd: dm-0: devmap already registered Jan 3 18:36:14 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 18:36:14 service103 multipathd: dm-1: add map (uevent) Jan 3 18:36:14 service103 kernel: sdc : very big device. try to use READ CAPACITY(16). Jan 3 18:36:14 service103 multipathd: dm-1: devmap already registered Jan 3 18:36:15 service103 kernel: SCSI device sdc: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:15 service103 multipathd: dm-2: add map (uevent) Jan 3 18:36:15 service103 kernel: sdc: Write Protect is off Jan 3 18:36:15 service103 multipathd: dm-2: devmap already registered Jan 3 18:36:15 service103 multipathd: sdg: add path (uevent) Jan 3 18:36:15 service103 kernel: SCSI device sdc: drive cache: write back w/ FUA Jan 3 18:36:15 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:96 10] Jan 3 18:36:15 service103 kernel: sdc: unknown partition table Jan 3 18:36:15 service103 multipathd: ddn6a-nbp6-ost42: event checker started Jan 3 18:36:15 service103 kernel: sd 2:0:0:11: Attached scsi disk sdc Jan 3 18:36:15 service103 multipathd: sdh: add path (uevent) Jan 3 18:36:15 service103 kernel: sd 2:0:0:11: Attached scsi generic sg6 type 0 Jan 3 18:36:15 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:112 10] Jan 3 18:36:15 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:15 service103 multipathd: ddn6a-nbp6-ost50: event checker started Jan 3 18:36:15 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:15 service103 multipathd: dm-3: add map (uevent) Jan 3 18:36:16 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 18:36:16 service103 multipathd: dm-3: devmap already registered Jan 3 18:36:16 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:16 service103 multipathd: dm-4: add map (uevent) Jan 3 18:36:16 service103 kernel: sdd: Write Protect is off Jan 3 18:36:16 service103 multipathd: dm-4: devmap already registered Jan 3 18:36:16 service103 multipathd: sdi: add path (uevent) Jan 3 18:36:16 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 18:36:16 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:128 10] Jan 3 18:36:16 service103 kernel: sdd : very big device. try to use READ CAPACITY(16). Jan 3 18:36:16 service103 multipathd: ddn6a-nbp6-ost58: event checker started Jan 3 18:36:16 service103 kernel: SCSI device sdd: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:16 service103 multipathd: sdj: add path (uevent) Jan 3 18:36:16 service103 kernel: sdd: Write Protect is off Jan 3 18:36:16 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:144 10] Jan 3 18:36:16 service103 multipathd: ddn6a-nbp6-ost66: event checker started Jan 3 18:36:17 service103 kernel: SCSI device sdd: drive cache: write back w/ FUA Jan 3 18:36:17 service103 multipathd: dm-5: add map (uevent) Jan 3 18:36:17 service103 kernel: sdd: unknown partition table Jan 3 18:36:17 service103 multipathd: dm-5: devmap already registered Jan 3 18:36:17 service103 kernel: sd 2:0:0:19: Attached scsi disk sdd Jan 3 18:36:17 service103 multipathd: sdk: add path (uevent) Jan 3 18:36:17 service103 kernel: sd 2:0:0:19: Attached scsi generic sg7 type 0 Jan 3 18:36:17 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:160 10] Jan 3 18:36:17 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:17 service103 multipathd: ddn6a-nbp6-ost74: event checker started Jan 3 18:36:17 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:17 service103 multipathd: dm-6: add map (uevent) Jan 3 18:36:17 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 18:36:17 service103 multipathd: dm-6: devmap already registered Jan 3 18:36:17 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:17 service103 multipathd: sdl: add path (uevent) Jan 3 18:36:17 service103 kernel: sde: Write Protect is off Jan 3 18:36:17 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:176 10] Jan 3 18:36:18 service103 multipathd: ddn6a-nbp6-ost82: event checker started Jan 3 18:36:18 service103 kernel: SCSI device sde: drive cache: write back w/ FUA Jan 3 18:36:18 service103 multipathd: sdm: add path (uevent) Jan 3 18:36:18 service103 kernel: sde : very big device. try to use READ CAPACITY(16). Jan 3 18:36:18 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:192 10] Jan 3 18:36:18 service103 kernel: SCSI device sde: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:18 service103 multipathd: ddn6a-nbp6-ost90: event checker started Jan 3 18:36:18 service103 kernel: sde: Write Protect is off Jan 3 18:36:18 service103 multipathd: dm-7: add map (uevent) Jan 3 18:36:18 service103 multipathd: dm-7: devmap already registered Jan 3 18:36:18 service103 kernel: SCSI device sde: drive cache: write back w/ FUA Jan 3 18:36:18 service103 multipathd: dm-8: add map (uevent) Jan 3 18:36:18 service103 kernel: sde: unknown partition table Jan 3 18:36:18 service103 multipathd: dm-8: devmap already registered Jan 3 18:36:18 service103 kernel: sd 2:0:0:27: Attached scsi disk sde Jan 3 18:36:18 service103 multipathd: dm-9: add map (uevent) Jan 3 18:36:18 service103 kernel: sd 2:0:0:27: Attached scsi generic sg8 type 0 Jan 3 18:36:18 service103 multipathd: dm-9: devmap already registered Jan 3 18:36:18 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:19 service103 multipathd: sdn: add path (uevent) Jan 3 18:36:19 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:19 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:208 10] Jan 3 18:36:19 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 18:36:19 service103 multipathd: ddn6a-nbp6-ost98: event checker started Jan 3 18:36:19 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:19 service103 multipathd: sdo: add path (uevent) Jan 3 18:36:19 service103 kernel: sdf: Write Protect is off Jan 3 18:36:19 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:224 10] Jan 3 18:36:19 service103 multipathd: ddn6a-nbp6-ost106: event checker started Jan 3 18:36:19 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 18:36:19 service103 multipathd: dm-10: add map (uevent) Jan 3 18:36:19 service103 kernel: sdf : very big device. try to use READ CAPACITY(16). Jan 3 18:36:19 service103 multipathd: dm-10: devmap already registered Jan 3 18:36:19 service103 xinetd[7314]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. Jan 3 18:36:19 service103 kernel: SCSI device sdf: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:19 service103 multipathd: sdp: add path (uevent) Jan 3 18:36:19 service103 xinetd[7314]: Started working: 2 available services Jan 3 18:36:19 service103 kernel: sdf: Write Protect is off Jan 3 18:36:19 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 1 1 round-robin 0 1 1 8:240 10] Jan 3 18:36:20 service103 multipathd: ddn6a-nbp6-ost114: event checker started Jan 3 18:36:20 service103 kernel: SCSI device sdf: drive cache: write back w/ FUA Jan 3 18:36:20 service103 multipathd: dm-11: add map (uevent) Jan 3 18:36:20 service103 kernel: sdf: unknown partition table Jan 3 18:36:20 service103 multipathd: dm-11: devmap already registered Jan 3 18:36:20 service103 kernel: sd 2:0:0:35: Attached scsi disk sdf Jan 3 18:36:20 service103 multipathd: dm-12: add map (uevent) Jan 3 18:36:20 service103 kernel: sd 2:0:0:35: Attached scsi generic sg9 type 0 Jan 3 18:36:20 service103 multipathd: dm-12: devmap already registered Jan 3 18:36:20 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:20 service103 multipathd: dm-13: add map (uevent) Jan 3 18:36:20 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:20 service103 multipathd: dm-13: devmap already registered Jan 3 18:36:21 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 18:36:21 service103 multipathd: dm-14: add map (uevent) Jan 3 18:36:21 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:21 service103 multipathd: dm-14: devmap already registered Jan 3 18:36:21 service103 kernel: sdg: Write Protect is off Jan 3 18:36:21 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 18:36:21 service103 kernel: sdg : very big device. try to use READ CAPACITY(16). Jan 3 18:36:21 service103 kernel: SCSI device sdg: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:22 service103 kernel: sdg: Write Protect is off Jan 3 18:36:22 service103 kernel: SCSI device sdg: drive cache: write back w/ FUA Jan 3 18:36:22 service103 kernel: sdg: unknown partition table Jan 3 18:36:22 service103 kernel: sd 2:0:0:43: Attached scsi disk sdg Jan 3 18:36:22 service103 kernel: sd 2:0:0:43: Attached scsi generic sg10 type 0 Jan 3 18:36:22 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:22 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:22 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 18:36:22 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:22 service103 kernel: sdh: Write Protect is off Jan 3 18:36:22 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 18:36:22 service103 kernel: sdh : very big device. try to use READ CAPACITY(16). Jan 3 18:36:22 service103 kernel: SCSI device sdh: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:22 service103 kernel: sdh: Write Protect is off Jan 3 18:36:23 service103 kernel: SCSI device sdh: drive cache: write back w/ FUA Jan 3 18:36:23 service103 kernel: sdh: unknown partition table Jan 3 18:36:23 service103 kernel: sd 2:0:0:51: Attached scsi disk sdh Jan 3 18:36:23 service103 kernel: sd 2:0:0:51: Attached scsi generic sg11 type 0 Jan 3 18:36:23 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:23 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:23 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 18:36:23 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:23 service103 kernel: sdi: Write Protect is off Jan 3 18:36:23 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 18:36:23 service103 kernel: sdi : very big device. try to use READ CAPACITY(16). Jan 3 18:36:23 service103 kernel: SCSI device sdi: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:23 service103 kernel: sdi: Write Protect is off Jan 3 18:36:24 service103 kernel: SCSI device sdi: drive cache: write back w/ FUA Jan 3 18:36:24 service103 kernel: sdi:<6>ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Jan 3 18:36:24 service103 kernel: unknown partition table Jan 3 18:36:24 service103 kernel: sd 2:0:0:59: Attached scsi disk sdi Jan 3 18:36:24 service103 kernel: sd 2:0:0:59: Attached scsi generic sg12 type 0 Jan 3 18:36:24 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:24 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:24 service103 kernel: sdj : very big device. try to use READ CAPACITY(16). Jan 3 18:36:24 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:24 service103 kernel: sdj: Write Protect is off Jan 3 18:36:24 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 18:36:24 service103 kernel: sdj : very big device. try to use READ CAPACITY(16). Jan 3 18:36:25 service103 kernel: SCSI device sdj: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:25 service103 kernel: sdj: Write Protect is off Jan 3 18:36:25 service103 kernel: SCSI device sdj: drive cache: write back w/ FUA Jan 3 18:36:25 service103 kernel: sdj: unknown partition table Jan 3 18:36:25 service103 kernel: sd 2:0:0:67: Attached scsi disk sdj Jan 3 18:36:25 service103 kernel: sd 2:0:0:67: Attached scsi generic sg13 type 0 Jan 3 18:36:25 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:25 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:25 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 18:36:25 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:25 service103 kernel: sdk: Write Protect is off Jan 3 18:36:25 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 18:36:26 service103 kernel: sdk : very big device. try to use READ CAPACITY(16). Jan 3 18:36:26 service103 kernel: SCSI device sdk: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:26 service103 kernel: sdk: Write Protect is off Jan 3 18:36:26 service103 kernel: SCSI device sdk: drive cache: write back w/ FUA Jan 3 18:36:26 service103 kernel: sdk: unknown partition table Jan 3 18:36:26 service103 kernel: sd 2:0:0:75: Attached scsi disk sdk Jan 3 18:36:26 service103 kernel: sd 2:0:0:75: Attached scsi generic sg14 type 0 Jan 3 18:36:26 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:26 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:26 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 18:36:26 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:26 service103 kernel: sdl: Write Protect is off Jan 3 18:36:27 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 18:36:27 service103 kernel: sdl : very big device. try to use READ CAPACITY(16). Jan 3 18:36:27 service103 kernel: SCSI device sdl: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:27 service103 kernel: sdl: Write Protect is off Jan 3 18:36:27 service103 kernel: SCSI device sdl: drive cache: write back w/ FUA Jan 3 18:36:27 service103 kernel: sdl: unknown partition table Jan 3 18:36:27 service103 kernel: sd 2:0:0:83: Attached scsi disk sdl Jan 3 18:36:27 service103 kernel: sd 2:0:0:83: Attached scsi generic sg15 type 0 Jan 3 18:36:27 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:27 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:27 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 18:36:27 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:27 service103 kernel: sdm: Write Protect is off Jan 3 18:36:28 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 18:36:28 service103 kernel: sdm : very big device. try to use READ CAPACITY(16). Jan 3 18:36:28 service103 kernel: SCSI device sdm: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:28 service103 kernel: sdm: Write Protect is off Jan 3 18:36:28 service103 kernel: SCSI device sdm: drive cache: write back w/ FUA Jan 3 18:36:28 service103 kernel: sdm: unknown partition table Jan 3 18:36:28 service103 kernel: sd 2:0:0:91: Attached scsi disk sdm Jan 3 18:36:28 service103 kernel: sd 2:0:0:91: Attached scsi generic sg16 type 0 Jan 3 18:36:28 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:28 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:28 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 18:36:28 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:29 service103 kernel: sdn: Write Protect is off Jan 3 18:36:29 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 18:36:29 service103 kernel: sdn : very big device. try to use READ CAPACITY(16). Jan 3 18:36:29 service103 kernel: SCSI device sdn: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:29 service103 kernel: sdn: Write Protect is off Jan 3 18:36:29 service103 kernel: SCSI device sdn: drive cache: write back w/ FUA Jan 3 18:36:29 service103 kernel: sdn: unknown partition table Jan 3 18:36:30 service103 kernel: sd 2:0:0:99: Attached scsi disk sdn Jan 3 18:36:30 service103 kernel: sd 2:0:0:99: Attached scsi generic sg17 type 0 Jan 3 18:36:30 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:30 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:30 service103 kernel: sdo : very big device. try to use READ CAPACITY(16). Jan 3 18:36:30 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:30 service103 kernel: sdo: Write Protect is off Jan 3 18:36:30 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 18:36:30 service103 kernel: sdo : very big device. try to use READ CAPACITY(16). Jan 3 18:36:30 service103 kernel: SCSI device sdo: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:30 service103 kernel: sdo: Write Protect is off Jan 3 18:36:30 service103 kernel: SCSI device sdo: drive cache: write back w/ FUA Jan 3 18:36:31 service103 kernel: sdo: unknown partition table Jan 3 18:36:31 service103 kernel: sd 2:0:0:107: Attached scsi disk sdo Jan 3 18:36:31 service103 kernel: sd 2:0:0:107: Attached scsi generic sg18 type 0 Jan 3 18:36:31 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:31 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:31 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 18:36:31 service103 kernel: SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:31 service103 kernel: sdp: Write Protect is off Jan 3 18:36:31 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 18:36:31 service103 kernel: sdp : very big device. try to use READ CAPACITY(16). Jan 3 18:36:31 service103 kernel: SCSI device sdp: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:31 service103 kernel: sdp: Write Protect is off Jan 3 18:36:31 service103 kernel: SCSI device sdp: drive cache: write back w/ FUA Jan 3 18:36:32 service103 kernel: sdp: unknown partition table Jan 3 18:36:32 service103 kernel: sd 2:0:0:115: Attached scsi disk sdp Jan 3 18:36:32 service103 kernel: sd 2:0:0:115: Attached scsi generic sg19 type 0 Jan 3 18:36:32 service103 kernel: ib_srp: ASYNC event= 17 on device= mlx4_1 Jan 3 18:36:32 service103 kernel: ib_srp: ASYNC event= 11 on device= mlx4_1 Jan 3 18:36:32 service103 kernel: ib_srp: ASYNC event= 9 on device= mlx4_1 Jan 3 18:36:36 service103 ntpdate[7352]: step time server 172.29.0.1 offset 0.008935 sec Jan 3 18:36:36 service103 ntpd[7377]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) Jan 3 18:36:36 service103 ntpd[7378]: precision = 1.000 usec Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface wildcard, ::#123 Disabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface lo, ::1#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface ib1, fe80::202:c903:f:9f84#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface eth0, fe80::230:48ff:fec4:4f0c#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface ib0, fe80::202:c903:f:9f83#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface lo, 127.0.0.1#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface eth0, 172.29.1.8#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface ib0, 10.150.25.157#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: Listening on interface ib1, 10.151.25.157#123 Enabled Jan 3 18:36:36 service103 ntpd[7378]: kernel time sync status 0040 Jan 3 18:36:38 service103 00-update-tempo-configs: rsync -a rsync://admin/tempo-configs/per-type/service/ . failed Jan 3 18:36:39 service103 leader-nodes-to-hosts-file: updating leader node entries in /etc/hosts... Jan 3 18:36:39 service103 kernel: scsi3 : SRP.T10:56980F0003C90200 Jan 3 18:36:39 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:39 service103 kernel: Type: RAID ANSI SCSI revision: 05 Jan 3 18:36:39 service103 kernel: scsi 3:0:0:0: Attached scsi generic sg20 type 12 Jan 3 18:36:39 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:39 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:39 service103 kernel: sd 3:0:0:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Jan 3 18:36:39 service103 kernel: sdq: Unit Not Ready, sense: Jan 3 18:36:39 service103 kernel: : Current: sense key: Unit Attention Jan 3 18:36:39 service103 kernel: Add. Sense: Reported luns data has changed Jan 3 18:36:39 service103 kernel: Jan 3 18:36:39 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 18:36:39 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:39 service103 multipathd: sdq: add path (uevent) Jan 3 18:36:39 service103 kernel: sdq: Write Protect is off Jan 3 18:36:39 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 18:36:39 service103 kernel: sdq : very big device. try to use READ CAPACITY(16). Jan 3 18:36:40 service103 kernel: SCSI device sdq: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:40 service103 ntpd[7378]: ntpd exiting on signal 15 Jan 3 18:36:40 service103 multipathd: ddn6a-nbp6-ost2: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:0 10 round-robin 0 1 1 8:16 10] Jan 3 18:36:40 service103 kernel: sdq: Write Protect is off Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 ntpd[7378]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 multipathd: sdr: add path (uevent) Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 logger: Adjusted sdq max_sectors_kb=4096 Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 logger: Adjusted sdr max_sectors_kb=4096 Jan 3 18:36:40 service103 logger: Adjusted blockdev Jan 3 18:36:40 service103 logger: Adjusted sds max_sectors_kb=4096 Jan 3 18:36:40 service103 multipathd: ddn6a-nbp6-ost10: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:16 10 round-robin 0 1 1 8:32 10] Jan 3 18:36:41 service103 kernel: SCSI device sdq: drive cache: write back w/ FUA Jan 3 18:36:41 service103 logger: Adjusted sdt max_sectors_kb=4096 Jan 3 18:36:41 service103 logger: Adjusted sdq scheduler=deadline Jan 3 18:36:41 service103 logger: Adjusted blockdev Jan 3 18:36:41 service103 logger: Adjusted sdu max_sectors_kb=4096 Jan 3 18:36:41 service103 logger: Adjusted sdr scheduler=deadline Jan 3 18:36:41 service103 logger: Adjusted sdv max_sectors_kb=4096 Jan 3 18:36:41 service103 logger: Adjusted blockdev Jan 3 18:36:41 service103 logger: Adjusted sds scheduler=deadline Jan 3 18:36:41 service103 kernel: sdq: unknown partition table Jan 3 18:36:41 service103 multipathd: sds: add path (uevent) Jan 3 18:36:41 service103 logger: Adjusted blockdev Jan 3 18:36:41 service103 logger: Adjusted sdt scheduler=deadline Jan 3 18:36:41 service103 logger: Adjected sdq timeout=280 Jan 3 18:36:41 service103 logger: Adjusted sdy max_sectors_kb=4096 Jan 3 18:36:41 service103 logger: Adjusted blockdev Jan 3 18:36:41 service103 logger: Adjusted sdu scheduler=deadline Jan 3 18:36:41 service103 logger: Adjected sdr timeout=280 Jan 3 18:36:41 service103 logger: Adjusted blockdev Jan 3 18:36:41 service103 logger: Adjusted sdv scheduler=deadline Jan 3 18:36:41 service103 logger: Adjusted sdz max_sectors_kb=4096 Jan 3 18:36:41 service103 logger: Adjected sds timeout=280 Jan 3 18:36:42 service103 kernel: sd 3:0:0:3: Attached scsi disk sdq Jan 3 18:36:42 service103 multipathd: ddn6a-nbp6-ost18: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:32 10 round-robin 0 1 1 8:48 10] Jan 3 18:36:42 service103 logger: Adjusted blockdev Jan 3 18:36:42 service103 logger: Adjusted sdaa max_sectors_kb=4096 Jan 3 18:36:42 service103 logger: Adjected sdt timeout=280 Jan 3 18:36:42 service103 logger: Adjusted blockdev Jan 3 18:36:42 service103 logger: Adjusted sdy scheduler=deadline Jan 3 18:36:42 service103 logger: Adjusted sdab max_sectors_kb=4096 Jan 3 18:36:42 service103 logger: Adjected sdu timeout=280 Jan 3 18:36:42 service103 logger: Adjusted sdac max_sectors_kb=4096 Jan 3 18:36:42 service103 logger: Adjected sdv timeout=280 Jan 3 18:36:42 service103 logger: Adjusted sdz scheduler=deadline Jan 3 18:36:42 service103 kernel: sd 3:0:0:3: Attached scsi generic sg21 type 0 Jan 3 18:36:42 service103 multipathd: sdt: add path (uevent) Jan 3 18:36:42 service103 logger: Adjusted sdad max_sectors_kb=4096 Jan 3 18:36:42 service103 logger: Adjusted sdaa scheduler=deadline Jan 3 18:36:42 service103 logger: Adjusted sdae max_sectors_kb=4096 Jan 3 18:36:42 service103 logger: Adjected sdy timeout=280 Jan 3 18:36:42 service103 logger: Adjusted sdab scheduler=deadline Jan 3 18:36:42 service103 logger: Adjusted sdac scheduler=deadline Jan 3 18:36:43 service103 logger: Adjected sdz timeout=280 Jan 3 18:36:43 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:43 service103 multipathd: ddn6a-nbp6-ost26: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:48 10 round-robin 0 1 1 8:64 10] Jan 3 18:36:43 service103 logger: Adjusted sdad scheduler=deadline Jan 3 18:36:43 service103 logger: Adjected sdaa timeout=280 Jan 3 18:36:43 service103 logger: Adjusted sdae scheduler=deadline Jan 3 18:36:43 service103 logger: Adjected sdab timeout=280 Jan 3 18:36:43 service103 logger: Adjected sdac timeout=280 Jan 3 18:36:43 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:43 service103 multipathd: sdu: add path (uevent) Jan 3 18:36:43 service103 logger: Adjected sdad timeout=280 Jan 3 18:36:43 service103 logger: Adjected sdae timeout=280 Jan 3 18:36:43 service103 kernel: sdr : very big device. try to use READ CAPACITY(16). Jan 3 18:36:43 service103 multipathd: ddn6a-nbp6-ost34: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:64 10 round-robin 0 1 1 8:80 10] Jan 3 18:36:43 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:43 service103 multipathd: dm-0: add map (uevent) Jan 3 18:36:44 service103 kernel: sdr: Write Protect is off Jan 3 18:36:44 service103 multipathd: dm-0: devmap already registered Jan 3 18:36:44 service103 multipathd: dm-1: add map (uevent) Jan 3 18:36:44 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 18:36:44 service103 multipathd: dm-1: devmap already registered Jan 3 18:36:44 service103 kernel: sdr : very big device. try to use READ CAPACITY(16). Jan 3 18:36:44 service103 multipathd: sdv: add path (uevent) Jan 3 18:36:44 service103 kernel: SCSI device sdr: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:44 service103 multipathd: ddn6a-nbp6-ost42: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:80 10 round-robin 0 1 1 8:96 10] Jan 3 18:36:44 service103 kernel: sdr: Write Protect is off Jan 3 18:36:44 service103 multipathd: sdw: add path (uevent) Jan 3 18:36:44 service103 multipathd: ddn6a-nbp6-ost50: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:96 10 round-robin 0 1 1 8:112 10] Jan 3 18:36:44 service103 kernel: SCSI device sdr: drive cache: write back w/ FUA Jan 3 18:36:44 service103 multipathd: dm-2: add map (uevent) Jan 3 18:36:44 service103 kernel: sdr: unknown partition table Jan 3 18:36:44 service103 multipathd: dm-2: devmap already registered Jan 3 18:36:44 service103 kernel: sd 3:0:0:11: Attached scsi disk sdr Jan 3 18:36:44 service103 multipathd: dm-3: add map (uevent) Jan 3 18:36:44 service103 kernel: sd 3:0:0:11: Attached scsi generic sg22 type 0 Jan 3 18:36:45 service103 multipathd: dm-3: devmap already registered Jan 3 18:36:45 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:45 service103 multipathd: sdx: add path (uevent) Jan 3 18:36:45 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:45 service103 multipathd: ddn6a-nbp6-ost58: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:112 10 round-robin 0 1 1 8:128 10] Jan 3 18:36:45 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 18:36:45 service103 multipathd: sdy: add path (uevent) Jan 3 18:36:45 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:45 service103 multipathd: ddn6a-nbp6-ost66: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:128 10 round-robin 0 1 1 8:144 10] Jan 3 18:36:45 service103 kernel: sds: Write Protect is off Jan 3 18:36:45 service103 multipathd: dm-4: add map (uevent) Jan 3 18:36:45 service103 multipathd: dm-4: devmap already registered Jan 3 18:36:45 service103 kernel: SCSI device sds: drive cache: write back w/ FUA Jan 3 18:36:45 service103 multipathd: sdz: add path (uevent) Jan 3 18:36:45 service103 kernel: sds : very big device. try to use READ CAPACITY(16). Jan 3 18:36:45 service103 multipathd: ddn6a-nbp6-ost74: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:144 10 round-robin 0 1 1 8:160 10] Jan 3 18:36:45 service103 kernel: SCSI device sds: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:45 service103 multipathd: dm-5: add map (uevent) Jan 3 18:36:46 service103 kernel: sds: Write Protect is off Jan 3 18:36:46 service103 multipathd: dm-5: devmap already registered Jan 3 18:36:46 service103 multipathd: sdaa: add path (uevent) Jan 3 18:36:46 service103 kernel: SCSI device sds: drive cache: write back w/ FUA Jan 3 18:36:46 service103 multipathd: ddn6a-nbp6-ost82: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:160 10 round-robin 0 1 1 8:176 10] Jan 3 18:36:46 service103 kernel: sds: unknown partition table Jan 3 18:36:46 service103 multipathd: dm-6: add map (uevent) Jan 3 18:36:46 service103 kernel: sd 3:0:0:19: Attached scsi disk sds Jan 3 18:36:46 service103 multipathd: dm-6: devmap already registered Jan 3 18:36:46 service103 kernel: sd 3:0:0:19: Attached scsi generic sg23 type 0 Jan 3 18:36:46 service103 multipathd: sdab: add path (uevent) Jan 3 18:36:46 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:46 service103 multipathd: ddn6a-nbp6-ost90: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:176 10 round-robin 0 1 1 8:192 10] Jan 3 18:36:46 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:46 service103 multipathd: dm-7: add map (uevent) Jan 3 18:36:46 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 18:36:46 service103 multipathd: dm-7: devmap already registered Jan 3 18:36:47 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:47 service103 multipathd: dm-8: add map (uevent) Jan 3 18:36:47 service103 kernel: sdt: Write Protect is off Jan 3 18:36:47 service103 multipathd: dm-8: devmap already registered Jan 3 18:36:47 service103 multipathd: sdac: add path (uevent) Jan 3 18:36:47 service103 kernel: SCSI device sdt: drive cache: write back w/ FUA Jan 3 18:36:47 service103 multipathd: ddn6a-nbp6-ost98: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:192 10 round-robin 0 1 1 8:208 10] Jan 3 18:36:47 service103 kernel: sdt : very big device. try to use READ CAPACITY(16). Jan 3 18:36:47 service103 multipathd: sdad: add path (uevent) Jan 3 18:36:47 service103 kernel: SCSI device sdt: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:47 service103 multipathd: ddn6a-nbp6-ost106: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:208 10 round-robin 0 1 1 8:224 10] Jan 3 18:36:47 service103 kernel: sdt: Write Protect is off Jan 3 18:36:47 service103 multipathd: dm-9: add map (uevent) Jan 3 18:36:47 service103 multipathd: dm-9: devmap already registered Jan 3 18:36:47 service103 kernel: SCSI device sdt: drive cache: write back w/ FUA Jan 3 18:36:47 service103 multipathd: dm-10: add map (uevent) Jan 3 18:36:48 service103 kernel: sdt: unknown partition table Jan 3 18:36:48 service103 multipathd: dm-10: devmap already registered Jan 3 18:36:48 service103 kernel: sd 3:0:0:27: Attached scsi disk sdt Jan 3 18:36:48 service103 multipathd: sdae: add path (uevent) Jan 3 18:36:48 service103 kernel: sd 3:0:0:27: Attached scsi generic sg24 type 0 Jan 3 18:36:48 service103 multipathd: ddn6a-nbp6-ost114: load table [0 15149826048 multipath 0 0 2 1 round-robin 0 1 1 65:224 10 round-robin 0 1 1 8:240 10] Jan 3 18:36:48 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:48 service103 multipathd: dm-11: add map (uevent) Jan 3 18:36:48 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:48 service103 multipathd: dm-11: devmap already registered Jan 3 18:36:48 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 18:36:48 service103 multipathd: dm-12: add map (uevent) Jan 3 18:36:48 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:48 service103 multipathd: dm-12: devmap already registered Jan 3 18:36:48 service103 kernel: sdu: Write Protect is off Jan 3 18:36:48 service103 multipathd: dm-13: add map (uevent) Jan 3 18:36:48 service103 multipathd: dm-13: devmap already registered Jan 3 18:36:48 service103 kernel: SCSI device sdu: drive cache: write back w/ FUA Jan 3 18:36:49 service103 multipathd: dm-14: add map (uevent) Jan 3 18:36:49 service103 kernel: sdu : very big device. try to use READ CAPACITY(16). Jan 3 18:36:49 service103 multipathd: dm-14: devmap already registered Jan 3 18:36:49 service103 kernel: SCSI device sdu: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:49 service103 kernel: sdu: Write Protect is off Jan 3 18:36:49 service103 kernel: SCSI device sdu: drive cache: write back w/ FUA Jan 3 18:36:49 service103 kernel: sdu: unknown partition table Jan 3 18:36:49 service103 kernel: sd 3:0:0:35: Attached scsi disk sdu Jan 3 18:36:49 service103 kernel: sd 3:0:0:35: Attached scsi generic sg25 type 0 Jan 3 18:36:49 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:49 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:49 service103 kernel: sdv : very big device. try to use READ CAPACITY(16). Jan 3 18:36:49 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:49 service103 kernel: sdv: Write Protect is off Jan 3 18:36:49 service103 kernel: SCSI device sdv: drive cache: write back w/ FUA Jan 3 18:36:50 service103 kernel: sdv : very big device. try to use READ CAPACITY(16). Jan 3 18:36:50 service103 kernel: SCSI device sdv: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:50 service103 kernel: sdv: Write Protect is off Jan 3 18:36:50 service103 kernel: SCSI device sdv: drive cache: write back w/ FUA Jan 3 18:36:50 service103 kernel: sdv: unknown partition table Jan 3 18:36:50 service103 kernel: sd 3:0:0:43: Attached scsi disk sdv Jan 3 18:36:50 service103 kernel: sd 3:0:0:43: Attached scsi generic sg26 type 0 Jan 3 18:36:50 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:50 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:50 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 18:36:50 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:50 service103 kernel: sdw: Write Protect is off Jan 3 18:36:51 service103 kernel: SCSI device sdw: drive cache: write back w/ FUA Jan 3 18:36:51 service103 kernel: sdw : very big device. try to use READ CAPACITY(16). Jan 3 18:36:51 service103 kernel: SCSI device sdw: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:51 service103 kernel: sdw: Write Protect is off Jan 3 18:36:51 service103 kernel: SCSI device sdw: drive cache: write back w/ FUA Jan 3 18:36:51 service103 kernel: sdw: unknown partition table Jan 3 18:36:51 service103 kernel: sd 3:0:0:51: Attached scsi disk sdw Jan 3 18:36:51 service103 kernel: sd 3:0:0:51: Attached scsi generic sg27 type 0 Jan 3 18:36:51 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:51 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:51 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 18:36:51 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:51 service103 kernel: sdx: Write Protect is off Jan 3 18:36:52 service103 kernel: SCSI device sdx: drive cache: write back w/ FUA Jan 3 18:36:52 service103 kernel: sdx : very big device. try to use READ CAPACITY(16). Jan 3 18:36:52 service103 kernel: SCSI device sdx: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:52 service103 kernel: sdx: Write Protect is off Jan 3 18:36:52 service103 kernel: SCSI device sdx: drive cache: write back w/ FUA Jan 3 18:36:52 service103 kernel: sdx: unknown partition table Jan 3 18:36:52 service103 kernel: sd 3:0:0:59: Attached scsi disk sdx Jan 3 18:36:52 service103 kernel: sd 3:0:0:59: Attached scsi generic sg28 type 0 Jan 3 18:36:52 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:52 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:52 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 18:36:52 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:53 service103 kernel: sdy: Write Protect is off Jan 3 18:36:53 service103 kernel: SCSI device sdy: drive cache: write back w/ FUA Jan 3 18:36:53 service103 kernel: sdy : very big device. try to use READ CAPACITY(16). Jan 3 18:36:53 service103 kernel: SCSI device sdy: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:53 service103 kernel: sdy: Write Protect is off Jan 3 18:36:53 service103 kernel: SCSI device sdy: drive cache: write back w/ FUA Jan 3 18:36:53 service103 kernel: sdy: unknown partition table Jan 3 18:36:54 service103 kernel: sd 3:0:0:67: Attached scsi disk sdy Jan 3 18:36:54 service103 kernel: sd 3:0:0:67: Attached scsi generic sg29 type 0 Jan 3 18:36:54 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:54 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:54 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 18:36:54 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:54 service103 kernel: sdz: Write Protect is off Jan 3 18:36:54 service103 kernel: SCSI device sdz: drive cache: write back w/ FUA Jan 3 18:36:54 service103 kernel: sdz : very big device. try to use READ CAPACITY(16). Jan 3 18:36:54 service103 kernel: SCSI device sdz: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:54 service103 kernel: sdz: Write Protect is off Jan 3 18:36:54 service103 kernel: SCSI device sdz: drive cache: write back w/ FUA Jan 3 18:36:54 service103 kernel: sdz: unknown partition table Jan 3 18:36:55 service103 kernel: sd 3:0:0:75: Attached scsi disk sdz Jan 3 18:36:55 service103 kernel: sd 3:0:0:75: Attached scsi generic sg30 type 0 Jan 3 18:36:55 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:55 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:55 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 18:36:55 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:55 service103 kernel: sdaa: Write Protect is off Jan 3 18:36:55 service103 kernel: SCSI device sdaa: drive cache: write back w/ FUA Jan 3 18:36:55 service103 kernel: sdaa : very big device. try to use READ CAPACITY(16). Jan 3 18:36:55 service103 kernel: SCSI device sdaa: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:55 service103 kernel: sdaa: Write Protect is off Jan 3 18:36:55 service103 kernel: SCSI device sdaa: drive cache: write back w/ FUA Jan 3 18:36:56 service103 kernel: sdaa: unknown partition table Jan 3 18:36:56 service103 kernel: sd 3:0:0:83: Attached scsi disk sdaa Jan 3 18:36:56 service103 kernel: sd 3:0:0:83: Attached scsi generic sg31 type 0 Jan 3 18:36:56 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:56 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:56 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 18:36:56 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:56 service103 kernel: sdab: Write Protect is off Jan 3 18:36:56 service103 kernel: SCSI device sdab: drive cache: write back w/ FUA Jan 3 18:36:56 service103 kernel: sdab : very big device. try to use READ CAPACITY(16). Jan 3 18:36:56 service103 kernel: SCSI device sdab: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:56 service103 kernel: sdab: Write Protect is off Jan 3 18:36:57 service103 kernel: SCSI device sdab: drive cache: write back w/ FUA Jan 3 18:36:57 service103 kernel: sdab: unknown partition table Jan 3 18:36:57 service103 kernel: sd 3:0:0:91: Attached scsi disk sdab Jan 3 18:36:57 service103 kernel: sd 3:0:0:91: Attached scsi generic sg32 type 0 Jan 3 18:36:57 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:57 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:57 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 18:36:57 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:57 service103 kernel: sdac: Write Protect is off Jan 3 18:36:57 service103 kernel: SCSI device sdac: drive cache: write back w/ FUA Jan 3 18:36:57 service103 kernel: sdac : very big device. try to use READ CAPACITY(16). Jan 3 18:36:57 service103 kernel: SCSI device sdac: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:57 service103 kernel: sdac: Write Protect is off Jan 3 18:36:58 service103 kernel: SCSI device sdac: drive cache: write back w/ FUA Jan 3 18:36:58 service103 kernel: sdac: unknown partition table Jan 3 18:36:58 service103 kernel: sd 3:0:0:99: Attached scsi disk sdac Jan 3 18:36:58 service103 kernel: sd 3:0:0:99: Attached scsi generic sg33 type 0 Jan 3 18:36:58 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:58 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:58 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 18:36:58 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:58 service103 kernel: sdad: Write Protect is off Jan 3 18:36:58 service103 kernel: SCSI device sdad: drive cache: write back w/ FUA Jan 3 18:36:58 service103 kernel: sdad : very big device. try to use READ CAPACITY(16). Jan 3 18:36:58 service103 kernel: SCSI device sdad: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:59 service103 kernel: sdad: Write Protect is off Jan 3 18:36:59 service103 kernel: SCSI device sdad: drive cache: write back w/ FUA Jan 3 18:36:59 service103 kernel: sdad: unknown partition table Jan 3 18:36:59 service103 kernel: sd 3:0:0:107: Attached scsi disk sdad Jan 3 18:36:59 service103 kernel: sd 3:0:0:107: Attached scsi generic sg34 type 0 Jan 3 18:36:59 service103 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Jan 3 18:36:59 service103 kernel: Type: Direct-Access ANSI SCSI revision: 05 Jan 3 18:36:59 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 18:36:59 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:36:59 service103 kernel: sdae: Write Protect is off Jan 3 18:36:59 service103 kernel: SCSI device sdae: drive cache: write back w/ FUA Jan 3 18:36:59 service103 kernel: sdae : very big device. try to use READ CAPACITY(16). Jan 3 18:37:00 service103 kernel: SCSI device sdae: 15149826048 512-byte hdwr sectors (7756711 MB) Jan 3 18:37:00 service103 kernel: sdae: Write Protect is off Jan 3 18:37:00 service103 kernel: SCSI device sdae: drive cache: write back w/ FUA Jan 3 18:37:00 service103 kernel: sdae: unknown partition table Jan 3 18:37:00 service103 kernel: sd 3:0:0:115: Attached scsi disk sdae Jan 3 18:37:00 service103 kernel: sd 3:0:0:115: Attached scsi generic sg35 type 0 Jan 3 18:37:03 service103 ntpdate[8149]: step time server 172.29.0.1 offset -0.014329 sec Jan 3 18:37:03 service103 ntpd[8560]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) Jan 3 18:37:03 service103 ntpd[8561]: precision = 1.000 usec Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface wildcard, ::#123 Disabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface lo, ::1#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface ib1, fe80::202:c903:f:9f84#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface eth0, fe80::230:48ff:fec4:4f0c#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface ib0, fe80::202:c903:f:9f83#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface lo, 127.0.0.1#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface eth0, 172.29.1.8#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface ib0, 10.150.25.157#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: Listening on interface ib1, 10.151.25.157#123 Enabled Jan 3 18:37:03 service103 ntpd[8561]: kernel time sync status 0040 Jan 3 18:37:20 service103 smartd[9667]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Jan 3 18:37:20 service103 smartd[9667]: Home page is http://smartmontools.sourceforge.net/ Jan 3 18:37:20 service103 smartd[9667]: Opened configuration file /etc/smartd.conf Jan 3 18:37:20 service103 smartd[9667]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/hdb, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/hdb, packet devices [this device CD/DVD] not SMART capable Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sda, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sda, Bad IEC (SMART) mode page, err=4, skip device Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdb, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list. Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdc, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list. Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdd, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdd, is SMART capable. Adding to "monitor" list. Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sde, opened Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sde, is SMART capable. Adding to "monitor" list. Jan 3 18:37:20 service103 smartd[9667]: Device: /dev/sdf, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdf, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdg, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdg, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdh, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdh, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdi, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdi, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdj, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdj, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdk, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdk, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdl, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdl, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdm, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdm, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdn, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdn, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdo, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdo, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdp, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdp, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdq, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdq, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdr, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sdr, is SMART capable. Adding to "monitor" list. Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sds, opened Jan 3 18:37:21 service103 smartd[9667]: Device: /dev/sds, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdt, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdt, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdu, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdu, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdv, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdv, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdw, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdw, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdx, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdx, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdy, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdy, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdz, opened Jan 3 18:37:22 service103 smartd[9667]: Device: /dev/sdz, is SMART capable. Adding to "monitor" list. Jan 3 18:37:22 service103 smartd[9667]: Monitoring 0 ATA and 25 SCSI devices Jan 3 18:37:22 service103 smartd[9681]: smartd has fork()ed into background mode. New PID=9681. Jan 3 18:39:45 service103 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Jan 3 18:39:46 service103 multipathd: dm-3: umount map (uevent) Jan 3 18:39:47 service103 kernel: Lustre: Lustre Version: 1.8.6.81 Jan 3 18:39:48 service103 kernel: Lustre: Build Version: lustre/scripts-1.8.6 Jan 3 18:39:48 service103 kernel: Lustre: Listener bound to ib1:10.151.25.157:987:mlx4_0 Jan 3 18:39:48 service103 multipathd: dm-13: umount map (uevent) Jan 3 18:39:49 service103 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Jan 3 18:39:49 service103 multipathd: dm-8: umount map (uevent) Jan 3 18:39:49 service103 kernel: Lustre: Added LNI 10.151.25.157@o2ib [8/64/0/180] Jan 3 18:39:49 service103 multipathd: dm-14: umount map (uevent) Jan 3 18:39:49 service103 kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Jan 3 18:39:49 service103 multipathd: dm-9: umount map (uevent) Jan 3 18:39:49 service103 kernel: init dynlocks cache Jan 3 18:39:50 service103 multipathd: dm-10: umount map (uevent) Jan 3 18:39:50 service103 kernel: ldiskfs created from ext4-2.6-rhel5 Jan 3 18:39:50 service103 multipathd: dm-2: umount map (uevent) Jan 3 18:39:50 service103 kernel: LDISKFS-fs (dm-3): recovery complete Jan 3 18:39:50 service103 multipathd: dm-6: umount map (uevent) Jan 3 18:39:51 service103 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode Jan 3 18:39:51 service103 multipathd: dm-11: umount map (uevent) Jan 3 18:39:51 service103 kernel: LDISKFS-fs (dm-13): recovery complete Jan 3 18:39:51 service103 multipathd: dm-0: umount map (uevent) Jan 3 18:39:51 service103 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Jan 3 18:39:51 service103 multipathd: dm-12: umount map (uevent) Jan 3 18:39:51 service103 kernel: JBD: barrier-based sync failed on dm-3-8 - disabling barriers Jan 3 18:39:51 service103 multipathd: dm-4: umount map (uevent) Jan 3 18:39:51 service103 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Jan 3 18:39:52 service103 multipathd: dm-1: umount map (uevent) Jan 3 18:39:52 service103 kernel: LDISKFS-fs (dm-8): recovery complete Jan 3 18:39:52 service103 multipathd: dm-7: umount map (uevent) Jan 3 18:39:52 service103 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode Jan 3 18:39:52 service103 multipathd: dm-5: umount map (uevent) Jan 3 18:39:52 service103 kernel: JBD: barrier-based sync failed on dm-8-8 - disabling barriers Jan 3 18:39:52 service103 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode Jan 3 18:39:52 service103 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Jan 3 18:39:52 service103 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode Jan 3 18:39:52 service103 kernel: Lustre: MGC10.151.25.163@o2ib: Reactivating import Jan 3 18:39:52 service103 kernel: Lustre: Filtering OBD driver; http://wiki.whamcloud.com/ Jan 3 18:39:52 service103 kernel: JBD: barrier-based sync failed on dm-3-8 - disabling barriers Jan 3 18:39:52 service103 kernel: Lustre: nbp6-OST001a: Now serving nbp6-OST001a on /dev/mapper/ddn6a-nbp6-ost26 with recovery enabled Jan 3 18:39:52 service103 kernel: LustreError: 10642:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST001a: unknown param writethrough=0 Jan 3 18:39:52 service103 kernel: LustreError: 10062:0:(filter.c:4022:filter_iocontrol()) aborting recovery for device nbp6-OST001a Jan 3 18:39:52 service103 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Jan 3 18:39:52 service103 kernel: Lustre: nbp6-OST006a: Now serving nbp6-OST006a on /dev/mapper/ddn6a-nbp6-ost106 with recovery enabled Jan 3 18:39:52 service103 kernel: LustreError: 10658:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST006a: unknown param writethrough=0 Jan 3 18:39:53 service103 kernel: LustreError: 10058:0:(filter.c:4022:filter_iocontrol()) aborting recovery for device nbp6-OST006a Jan 3 18:39:53 service103 kernel: JBD: barrier-based sync failed on dm-8-8 - disabling barriers Jan 3 18:39:53 service103 kernel: LDISKFS-fs (dm-14): recovery complete Jan 3 18:39:53 service103 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Jan 3 18:39:53 service103 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Jan 3 18:39:53 service103 kernel: LDISKFS-fs (dm-9): recovery complete Jan 3 18:39:53 service103 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Jan 3 18:39:54 service103 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Jan 3 18:39:54 service103 kernel: LDISKFS-fs (dm-10): recovery complete Jan 3 18:39:54 service103 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Jan 3 18:39:54 service103 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Jan 3 18:39:54 service103 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Jan 3 18:39:55 service103 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Jan 3 18:39:55 service103 kernel: Lustre: nbp6-OST0072: Now serving nbp6-OST0072 on /dev/mapper/ddn6a-nbp6-ost114 with recovery enabled Jan 3 18:39:55 service103 kernel: Lustre: Skipped 1 previous similar message Jan 3 18:39:55 service103 kernel: LustreError: 10814:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST0072: unknown param writethrough=0 Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Jan 3 18:39:55 service103 kernel: LustreError: 10814:0:(obd_config.c:1011:class_process_proc_param()) Skipped 1 previous similar message Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-2): recovery complete Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode Jan 3 18:39:55 service103 kernel: JBD: barrier-based sync failed on dm-2-8 - disabling barriers Jan 3 18:39:55 service103 kernel: LustreError: 10712:0:(filter.c:4022:filter_iocontrol()) aborting recovery for device nbp6-OST0072 Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-6): recovery complete Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Jan 3 18:39:55 service103 kernel: LustreError: 10712:0:(filter.c:4022:filter_iocontrol()) Skipped 1 previous similar message Jan 3 18:39:55 service103 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Jan 3 18:39:55 service103 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Jan 3 18:39:55 service103 kernel: LDISKFS-fs (dm-11): recovery complete Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-11): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: JBD: barrier-based sync failed on dm-11-8 - disabling barriers Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-11): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: JBD: barrier-based sync failed on dm-2-8 - disabling barriers Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-0): recovery complete Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: JBD: barrier-based sync failed on dm-0-8 - disabling barriers Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-12): recovery complete Jan 3 18:39:56 service103 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode Jan 3 18:39:56 service103 kernel: JBD: barrier-based sync failed on dm-12-8 - disabling barriers Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-4): recovery complete Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: JBD: barrier-based sync failed on dm-4-8 - disabling barriers Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-1): recovery complete Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: JBD: barrier-based sync failed on dm-1-8 - disabling barriers Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-7): recovery complete Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode Jan 3 18:39:57 service103 kernel: JBD: barrier-based sync failed on dm-7-8 - disabling barriers Jan 3 18:39:58 service103 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode Jan 3 18:39:58 service103 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode Jan 3 18:39:58 service103 kernel: JBD: barrier-based sync failed on dm-11-8 - disabling barriers Jan 3 18:39:58 service103 kernel: JBD: barrier-based sync failed on dm-0-8 - disabling barriers Jan 3 18:39:58 service103 kernel: JBD: barrier-based sync failed on dm-12-8 - disabling barriers Jan 3 18:39:58 service103 kernel: JBD: barrier-based sync failed on dm-4-8 - disabling barriers Jan 3 18:39:58 service103 kernel: Lustre: nbp6-OST0022: Now serving nbp6-OST0022 on /dev/mapper/ddn6a-nbp6-ost34 with recovery enabled Jan 3 18:39:58 service103 kernel: Lustre: Skipped 7 previous similar messages Jan 3 18:39:58 service103 kernel: LustreError: 11214:0:(obd_config.c:1011:class_process_proc_param()) nbp6-OST0022: unknown param writethrough=0 Jan 3 18:39:58 service103 kernel: LustreError: 11214:0:(obd_config.c:1011:class_process_proc_param()) Skipped 7 previous similar messages Jan 3 18:39:58 service103 kernel: LustreError: 10909:0:(filter.c:4022:filter_iocontrol()) aborting recovery for device nbp6-OST0022 Jan 3 18:39:58 service103 kernel: LustreError: 10909:0:(filter.c:4022:filter_iocontrol()) Skipped 7 previous similar messages Jan 3 18:39:58 service103 kernel: JBD: barrier-based sync failed on dm-1-8 - disabling barriers Jan 3 18:39:59 service103 kernel: JBD: barrier-based sync failed on dm-7-8 - disabling barriers Jan 3 18:39:59 service103 kernel: LDISKFS-fs (dm-5): recovery complete Jan 3 18:39:59 service103 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Jan 3 18:39:59 service103 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Jan 3 18:39:59 service103 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Jan 3 18:39:59 service103 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Jan 3 18:40:00 service103 kernel: Lustre: nbp6-OST000a: received MDS connection from 10.151.25.163@o2ib Jan 3 18:40:00 service103 kernel: Lustre: 10368:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST000a: deleting orphan objects from 278851 to 278881, orphan objids won't be reused any more. Jan 3 18:40:01 service103 kernel: Lustre: nbp6-OST0002: received MDS connection from 10.151.25.163@o2ib Jan 3 18:40:01 service103 kernel: Lustre: 10406:0:(filter.c:3126:filter_destroy_precreated()) nbp6-OST0042: deleting orphan objects from 278723 to 278753, orphan objids won't be reused any more. Jan 3 18:40:01 service103 kernel: Lustre: Skipped 11 previous similar messages Jan 3 18:40:16 service103 ntpd[8561]: synchronized to 172.29.0.1, stratum 3 Jan 3 18:44:34 service103 kernel: Lustre: nbp6-OST0022: haven't heard from client c9ae4655-cf6b-a4a2-b058-e840639bbabf (at 10.151.61.100@o2ib) in 156 seconds. I think it's dead, and I am evicting it. Jan 3 18:44:34 service103 kernel: Lustre: nbp6-OST0022: haven't heard from client dd7b2405-cf3d-ce05-f6db-e79b9f920b33 (at 10.151.21.165@o2ib) in 156 seconds. I think it's dead, and I am evicting it. Jan 3 18:44:34 service103 kernel: LustreError: 10417:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109c8f93400 x1389019376657895/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645239 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:44:34 service103 kernel: LustreError: 10492:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109c84f8000 x1386973324455838/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645239 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:44:34 service103 kernel: LustreError: 10439:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109c3bf9000 x1389019258689621/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645239 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:44:34 service103 kernel: LustreError: 10439:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jan 3 18:44:36 service103 kernel: LustreError: 10424:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109b90eec00 x1386974066397538/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645240 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:44:36 service103 kernel: LustreError: 10424:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Jan 3 18:44:38 service103 kernel: LustreError: 10476:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109a4b9b800 x1389019361065759/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645243 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:44:38 service103 kernel: LustreError: 10476:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages Jan 3 18:47:27 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 4c9ab2f5-e996-cf29-481b-2e244a808193 (at 10.151.32.83@o2ib) in 153 seconds. I think it's dead, and I am evicting it. Jan 3 18:47:27 service103 kernel: Lustre: Skipped 82 previous similar messages Jan 3 18:47:31 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client 4c9ab2f5-e996-cf29-481b-2e244a808193 (at 10.151.32.83@o2ib) in 157 seconds. I think it's dead, and I am evicting it. Jan 3 18:47:31 service103 kernel: Lustre: Skipped 3 previous similar messages Jan 3 18:47:33 service103 kernel: LustreError: 10491:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81095e48b800 x1386959295206471/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325645418 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 18:47:33 service103 kernel: LustreError: 10491:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 44 previous similar messages Jan 3 18:57:40 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.2.120@o2ib [old ver: 12, new ver: 12] Jan 3 18:57:40 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.2.135@o2ib [old ver: 12, new ver: 12] Jan 3 18:57:41 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.15.54@o2ib [old ver: 12, new ver: 12] Jan 3 18:57:41 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 18:57:43 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.0.43@o2ib [old ver: 12, new ver: 12] Jan 3 18:58:25 service103 ntpd[8561]: kernel time sync enabled 0001 Jan 3 18:58:56 service103 kernel: Lustre: nbp6-OST002a: haven't heard from client 0aa05315-4b39-efbf-ef47-7fa39df96e30 (at 10.151.22.62@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 18:58:57 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client ba529cf1-bb34-b3ee-b6a3-2659931b54c9 (at 10.151.0.179@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 18:58:57 service103 kernel: Lustre: Skipped 2549 previous similar messages Jan 3 19:00:12 service103 kernel: Lustre: nbp6-OST002a: haven't heard from client f717dc9f-9712-70c4-0158-7f8cd458cc7b (at 10.151.13.98@o2ib) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 19:00:12 service103 kernel: Lustre: Skipped 1274 previous similar messages Jan 3 19:00:13 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client f717dc9f-9712-70c4-0158-7f8cd458cc7b (at 10.151.13.98@o2ib) in 153 seconds. I think it's dead, and I am evicting it. Jan 3 19:00:13 service103 kernel: Lustre: Skipped 19 previous similar messages Jan 3 19:00:16 service103 kernel: LustreError: 10492:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810a2afdf000 x1390040431201350/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1325646181 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 3 19:00:16 service103 kernel: LustreError: 10492:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 13 previous similar messages Jan 3 19:15:27 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.47@o2ib [old ver: 12, new ver: 12] Jan 3 19:15:27 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 19:15:29 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.71@o2ib [old ver: 12, new ver: 12] Jan 3 19:15:29 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 2 previous similar messages Jan 3 19:15:30 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.118@o2ib [old ver: 12, new ver: 12] Jan 3 19:15:30 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 19:15:37 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.19@o2ib [old ver: 12, new ver: 12] Jan 3 19:15:37 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 19:17:24 service103 kernel: Lustre: nbp6-OST0012: haven't heard from client 9f98b43a-fd30-4775-2248-98b57cf73f34 (at 10.151.48.48@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 19:17:24 service103 kernel: Lustre: Skipped 9 previous similar messages Jan 3 19:17:25 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.119@o2ib [old ver: 12, new ver: 12] Jan 3 19:17:25 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 19:18:40 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client beed5ef6-ce49-7ccb-271a-e4093b269466 (at 10.151.48.119@o2ib) in 200 seconds. I think it's dead, and I am evicting it. Jan 3 19:18:40 service103 kernel: Lustre: Skipped 224 previous similar messages Jan 3 19:37:03 service103 ntpd[8561]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 19:57:17 service103 kernel: Lustre: 3353:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.17.76@o2ib [old ver: 12, new ver: 12] Jan 3 19:59:08 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 6af0ccc9-8db4-8c7b-25fe-ac88dafb1a8d (at 10.151.17.76@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 19:59:08 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 20:16:11 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.68@o2ib [old ver: 12, new ver: 12] Jan 3 20:16:11 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.71@o2ib [old ver: 12, new ver: 12] Jan 3 20:16:11 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.49@o2ib [old ver: 12, new ver: 12] Jan 3 20:16:11 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 20:18:10 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 0ec4c502-0ef8-ae64-7562-6f0a94f6fc98 (at 10.151.48.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:18:10 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 20:18:10 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 7a35ed1e-1212-36c1-1120-9f5816a28919 (at 10.151.48.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:22:43 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.56@o2ib [old ver: 12, new ver: 12] Jan 3 20:22:43 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 2 previous similar messages Jan 3 20:22:55 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.68@o2ib [old ver: 12, new ver: 12] Jan 3 20:22:55 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 5 previous similar messages Jan 3 20:23:01 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.117@o2ib [old ver: 12, new ver: 12] Jan 3 20:23:01 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 2 previous similar messages Jan 3 20:24:47 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 824a6251-f93e-a95e-1ee2-5fcaf40898d1 (at 10.151.48.68@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:24:47 service103 kernel: Lustre: Skipped 118 previous similar messages Jan 3 20:25:43 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.48.58@o2ib [old ver: 12, new ver: 12] Jan 3 20:27:42 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 61c8db4b-f9ac-43eb-ad17-01ab6e13dc5e (at 10.151.48.48@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:27:42 service103 kernel: Lustre: Skipped 149 previous similar messages Jan 3 20:37:03 service103 ntpd[8561]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 20:38:02 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.43.118@o2ib [old ver: 12, new ver: 12] Jan 3 20:38:02 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 20:38:05 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.95@o2ib [old ver: 12, new ver: 12] Jan 3 20:38:05 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 20:40:01 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client 34cccea5-0546-0b90-1c7a-257124012ddc (at 10.151.45.54@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:40:01 service103 kernel: Lustre: Skipped 59 previous similar messages Jan 3 20:41:17 service103 kernel: Lustre: nbp6-OST001a: haven't heard from client 59e01056-bd1b-adbb-f999-20770b7eb1fb (at 10.151.45.61@o2ib) in 177 seconds. I think it's dead, and I am evicting it. Jan 3 20:41:17 service103 kernel: Lustre: Skipped 209 previous similar messages Jan 3 20:42:56 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.119@o2ib [old ver: 12, new ver: 12] Jan 3 20:42:56 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 8 previous similar messages Jan 3 20:44:40 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client ede28624-7b89-5340-2e12-af2869eabdd6 (at 10.151.50.120@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:44:40 service103 kernel: Lustre: Skipped 209 previous similar messages Jan 3 20:47:24 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.38.151@o2ib [old ver: 12, new ver: 12] Jan 3 20:47:24 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 20:49:14 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 25d86ef1-8734-a0d0-5e69-f4f49a0b3e5a (at 10.151.38.151@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:49:14 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 20:50:57 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.43.104@o2ib [old ver: 12, new ver: 12] Jan 3 20:52:51 service103 kernel: Lustre: nbp6-OST0022: haven't heard from client 2b089436-0776-d7de-1b20-c6998b670ae4 (at 10.151.44.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 20:52:51 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 20:52:55 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.30@o2ib [old ver: 12, new ver: 12] Jan 3 20:52:55 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 14 previous similar messages Jan 3 20:54:07 service103 kernel: Lustre: nbp6-OST000a: haven't heard from client 11ef7617-70d9-da03-be46-3e98081c9d0c (at 10.151.44.30@o2ib) in 188 seconds. I think it's dead, and I am evicting it. Jan 3 20:54:07 service103 kernel: Lustre: Skipped 224 previous similar messages Jan 3 20:54:54 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.174@o2ib [old ver: 12, new ver: 12] Jan 3 20:54:54 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 8 previous similar messages Jan 3 20:55:23 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client da3bc561-d622-f9e7-733f-97bbbc9f5bfc (at 10.151.50.174@o2ib) in 152 seconds. I think it's dead, and I am evicting it. Jan 3 20:55:23 service103 kernel: Lustre: Skipped 224 previous similar messages Jan 3 20:56:39 service103 kernel: Lustre: nbp6-OST004a: haven't heard from client 1f6cacdb-c133-1fa7-d6fc-b5297d6a8fce (at 10.151.50.173@o2ib) in 218 seconds. I think it's dead, and I am evicting it. Jan 3 20:56:39 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 21:02:48 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.129@o2ib [old ver: 12, new ver: 12] Jan 3 21:02:48 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 21:04:50 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client b01d84f6-a6e1-ee70-00f7-5cd74a437aed (at 10.151.50.63@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 21:04:50 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 21:37:03 service103 ntpd[8561]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 21:37:51 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.45.54@o2ib [old ver: 12, new ver: 12] Jan 3 21:37:51 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 7 previous similar messages Jan 3 21:39:58 service103 kernel: Lustre: nbp6-OST0072: haven't heard from client 4f14fb95-2919-ac85-2920-85775138002e (at 10.151.45.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 21:39:58 service103 kernel: Lustre: Skipped 119 previous similar messages Jan 3 21:40:56 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.45.60@o2ib [old ver: 12, new ver: 12] Jan 3 21:40:56 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 9 previous similar messages Jan 3 21:42:54 service103 kernel: Lustre: nbp6-OST0062: haven't heard from client f11b5878-238f-548e-4021-fb0fefe9c153 (at 10.151.45.60@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 21:42:54 service103 kernel: Lustre: Skipped 149 previous similar messages Jan 3 21:50:54 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.17@o2ib [old ver: 12, new ver: 12] Jan 3 21:50:54 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 21:52:53 service103 kernel: Lustre: nbp6-OST006a: haven't heard from client b8c488ad-30e7-9b59-efb2-0c0bbee22a58 (at 10.151.44.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 21:52:53 service103 kernel: Lustre: Skipped 89 previous similar messages Jan 3 21:54:41 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.95@o2ib [old ver: 12, new ver: 12] Jan 3 21:54:41 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 5 previous similar messages Jan 3 21:56:09 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 3bb35469-c0e5-7eac-29c2-4d476bd1fdf1 (at 10.151.43.103@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 21:56:09 service103 kernel: Lustre: Skipped 149 previous similar messages Jan 3 21:57:25 service103 kernel: Lustre: nbp6-OST004a: haven't heard from client 0e3a1dfa-df8c-0cb2-ddfe-4f752fe797f0 (at 10.151.45.58@o2ib) in 163 seconds. I think it's dead, and I am evicting it. Jan 3 21:57:25 service103 kernel: Lustre: Skipped 239 previous similar messages Jan 3 21:59:02 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.91@o2ib [old ver: 12, new ver: 12] Jan 3 21:59:02 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 18 previous similar messages Jan 3 21:59:57 service103 kernel: Lustre: nbp6-OST0012: haven't heard from client 03a6200e-0e59-9c1d-aedb-74bebee0d0c2 (at 10.151.50.120@o2ib) in 184 seconds. I think it's dead, and I am evicting it. Jan 3 21:59:57 service103 kernel: Lustre: Skipped 179 previous similar messages Jan 3 22:07:14 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 5e06e20c-865e-1c9e-cfaa-512805cca550 (at 10.151.44.135@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 22:07:14 service103 kernel: Lustre: Skipped 89 previous similar messages Jan 3 22:08:10 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.18@o2ib [old ver: 12, new ver: 12] Jan 3 22:08:10 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 12 previous similar messages Jan 3 22:29:41 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.19@o2ib [old ver: 12, new ver: 12] Jan 3 22:29:41 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 22:32:02 service103 kernel: Lustre: nbp6-OST0032: haven't heard from client 5feda35d-22fa-5267-1b3e-2bbc7f5c0c5a (at 10.151.44.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 22:32:02 service103 kernel: Lustre: Skipped 179 previous similar messages Jan 3 22:33:18 service103 kernel: Lustre: nbp6-OST005a: haven't heard from client ccf58f09-0f30-e2ed-a34d-abb9049d3287 (at 10.151.2.65@o2ib) in 160 seconds. I think it's dead, and I am evicting it. Jan 3 22:33:18 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 22:37:03 service103 ntpd[8561]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 22:39:39 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.17@o2ib [old ver: 12, new ver: 12] Jan 3 22:39:39 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 1 previous similar message Jan 3 22:41:40 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client fd6100a9-a5a6-5969-8fb0-fd7d46b3d574 (at 10.151.44.28@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 22:41:40 service103 kernel: Lustre: Skipped 29 previous similar messages Jan 3 22:45:46 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.30@o2ib [old ver: 12, new ver: 12] Jan 3 22:45:46 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 22:47:45 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client a1cc6ee1-0258-f570-b5ef-f4f779a310f5 (at 10.151.50.87@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 22:47:45 service103 kernel: Lustre: Skipped 59 previous similar messages Jan 3 23:06:09 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.44.64@o2ib [old ver: 12, new ver: 12] Jan 3 23:06:09 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 27 previous similar messages Jan 3 23:08:27 service103 kernel: Lustre: nbp6-OST002a: haven't heard from client 397611ed-96d1-e0c6-419f-502dd2ae8925 (at 10.151.44.64@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:08:27 service103 kernel: Lustre: Skipped 419 previous similar messages Jan 3 23:11:49 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.164@o2ib [old ver: 12, new ver: 12] Jan 3 23:11:49 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 3 previous similar messages Jan 3 23:13:51 service103 kernel: Lustre: nbp6-OST002a: haven't heard from client bfa52c6d-e23a-b5f3-4adc-4e2e18b19b5e (at 10.151.50.131@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:13:51 service103 kernel: Lustre: Skipped 59 previous similar messages Jan 3 23:33:08 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.43.142@o2ib [old ver: 12, new ver: 12] Jan 3 23:33:08 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 11 previous similar messages Jan 3 23:35:30 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client ca582421-5d44-21ae-367b-c3d91a65f575 (at 10.151.43.142@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:35:30 service103 kernel: Lustre: Skipped 179 previous similar messages Jan 3 23:37:03 service103 ntpd[8561]: can't open /var/lib/ntp/drift/ntp.drift.TEMP: Permission denied Jan 3 23:38:35 service103 kernel: Lustre: 3355:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.42.176@o2ib [old ver: 12, new ver: 12] Jan 3 23:40:37 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client e68f7e2c-3de4-846b-6d2d-49a64759f1c5 (at 10.151.43.142@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:40:37 service103 kernel: Lustre: Skipped 14 previous similar messages Jan 3 23:42:23 service103 kernel: Lustre: 3350:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.50.129@o2ib [old ver: 12, new ver: 12] Jan 3 23:42:23 service103 kernel: Lustre: 3350:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 4 previous similar messages Jan 3 23:44:17 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 1757314b-fb5b-33ed-bb1f-44aa08b776a1 (at 10.151.42.176@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:44:17 service103 kernel: Lustre: Skipped 74 previous similar messages Jan 3 23:51:38 service103 kernel: Lustre: 3350:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Conn stale 10.151.43.142@o2ib [old ver: 12, new ver: 12] Jan 3 23:51:38 service103 kernel: Lustre: 3350:0:(o2iblnd_cb.c:2245:kiblnd_passive_connect()) Skipped 13 previous similar messages Jan 3 23:53:53 service103 kernel: Lustre: nbp6-OST0002: haven't heard from client 9a1cd84e-1b53-76d1-1219-28939426f3cb (at 10.151.43.142@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 3 23:53:53 service103 kernel: Lustre: Skipped 209 previous similar messages