Oct  7 00:02:34 nbp1-oss8 pcp-pmie[4620]: High 1-minute load average 158load@nbp1-oss8
Oct  7 00:05:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.59.104@o2ib [old ver: 12, new ver: 12]
Oct  7 00:05:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 00:05:55 nbp1-oss8 kernel: Lustre: 65702:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 7bd723bf-cd54-37e9-86cf-2d06487a5eac@10.151.59.104@o2ib t0 exp (null) cur 1349593555 last 0
Oct  7 00:05:55 nbp1-oss8 kernel: Lustre: 65702:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 12 previous similar messages
Oct  7 00:10:02 nbp1-oss8 syslog-ng[3346]: Log statistics; processed='destination(d_messages)=33861', processed='destination(d_syslog)=464957', processed='src.internal(src#0)=239', stamp='src.internal(src#0)=1349565002', processed='center(queued)=0', processed='destination(acpid)=0', processed='destination(sec-logger)=0', processed='destination(uselog)=0', processed='destination(firewall)=0', dropped='dst.udp(sec-logger#0,10.151.0.15:514)=0', processed='dst.udp(sec-logger#0,10.151.0.15:514)=0', stored='dst.udp(sec-logger#0,10.151.0.15:514)=0', processed='global(payload_reallocs)=0', processed='destination(mail)=159', processed='destination(netmgm)=0', processed='destination(console)=30787', processed='destination(cron)=138823', processed='global(msg_clones)=0', processed='destination(d_warn)=30950', processed='global(sdata_updates)=0', processed='destination(localmessages)=33', processed='center(received)=0', processed='destination(loghost)=464462', processed='source(src)=465116', dropped='dst.udp(loghost#0,admin:514)=0', processed='dst.udp(loghost#0,admin:514)=464462', stored='dst.udp(loghost#0,admin:514)=0'
Oct  7 00:18:02 nbp1-oss8 syslog-ng[3346]: Syslog connection established; fd='13', server='AF_INET(172.29.0.1:514)', local='AF_INET(0.0.0.0:0)'
Oct  7 00:18:02 nbp1-oss8 syslog-ng[3346]: Syslog connection established; fd='14', server='AF_INET(10.151.0.15:514)', local='AF_INET(0.0.0.0:0)'
Oct  7 00:18:02 nbp1-oss8 syslog-ng[3346]: Configuration reload request received, reloading configuration;
Oct  7 00:24:54 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.15.105@o2ib [old ver: 12, new ver: 12]
Oct  7 00:24:54 nbp1-oss8 kernel: Lustre: 65320:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from d195cec6-6fde-6599-a6e5-5ecf712e035a@10.151.15.105@o2ib t0 exp (null) cur 1349594694 last 0
Oct  7 00:24:54 nbp1-oss8 kernel: Lustre: 65320:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 13 previous similar messages
Oct  7 00:24:54 nbp1-oss8 kernel: Lustre: 1903:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.17.111@o2ib [old ver: 12, new ver: 12]
Oct  7 00:24:54 nbp1-oss8 kernel: Lustre: 65741:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from ca511cff-5991-3df4-a9d2-ecd4e41a476e@10.151.17.111@o2ib t0 exp (null) cur 1349594694 last 0
Oct  7 00:24:55 nbp1-oss8 kernel: Lustre: 65741:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 50 previous similar messages
Oct  7 00:24:57 nbp1-oss8 kernel: Lustre: 1895:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.12.75@o2ib [old ver: 12, new ver: 12]
Oct  7 00:24:57 nbp1-oss8 kernel: Lustre: 1895:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 5 previous similar messages
Oct  7 00:24:57 nbp1-oss8 kernel: Lustre: 65788:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 561093ca-8646-8615-4ae8-bcd0dc4852c4@10.151.12.75@o2ib t0 exp (null) cur 1349594697 last 0
Oct  7 00:24:57 nbp1-oss8 kernel: Lustre: 65788:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 41 previous similar messages
Oct  7 00:25:03 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.11.89@o2ib [old ver: 12, new ver: 12]
Oct  7 00:25:03 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 00:25:03 nbp1-oss8 kernel: Lustre: 65999:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 678b3806-04cd-cffb-a788-59a21d3d8ee2@10.151.11.89@o2ib t0 exp (null) cur 1349594703 last 0
Oct  7 00:25:03 nbp1-oss8 kernel: Lustre: 65999:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 39 previous similar messages
Oct  7 00:25:05 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.12.68@o2ib [old ver: 12, new ver: 12]
Oct  7 00:25:05 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 4 previous similar messages
Oct  7 00:25:05 nbp1-oss8 kernel: Lustre: 65742:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 10c2e350-8d2d-a3f0-4c69-5e2814bfe390@10.151.12.68@o2ib t0 exp (null) cur 1349594705 last 0
Oct  7 00:25:05 nbp1-oss8 kernel: Lustre: 65742:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 69 previous similar messages
Oct  7 00:25:13 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.12.70@o2ib [old ver: 12, new ver: 12]
Oct  7 00:25:13 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 6 previous similar messages
Oct  7 00:25:13 nbp1-oss8 kernel: Lustre: 65731:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 6cc584c2-be5e-7da4-fb5b-8d73b234ede2@10.151.12.70@o2ib t0 exp (null) cur 1349594713 last 0
Oct  7 00:25:13 nbp1-oss8 kernel: Lustre: 65731:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 86 previous similar messages
Oct  7 00:26:08 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client d35caa77-00e7-5733-51a2-3e2c574e60d1 (at 10.151.15.105@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018db3a800, cur 1349594768 expire 1349594618 last 1349594541
Oct  7 00:26:08 nbp1-oss8 kernel: Lustre: Skipped 2 previous similar messages
Oct  7 00:26:09 nbp1-oss8 kernel: Lustre: nbp1-OST0067: haven't heard from client c92b90aa-318d-817d-86ab-42ce18cc8cd2 (at 10.151.11.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e222c3000, cur 1349594769 expire 1349594619 last 1349594542
Oct  7 00:26:09 nbp1-oss8 kernel: Lustre: Skipped 91 previous similar messages
Oct  7 01:00:07 nbp1-oss8 kernel: Lustre: 1888:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.16.193@o2ib [old ver: 12, new ver: 12]
Oct  7 01:00:07 nbp1-oss8 kernel: Lustre: 65643:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 37805b1e-0729-6b95-cbef-91a1886ab5fa@10.151.16.193@o2ib t0 exp (null) cur 1349596807 last 0
Oct  7 01:00:07 nbp1-oss8 kernel: Lustre: 65336:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 37805b1e-0729-6b95-cbef-91a1886ab5fa@10.151.16.193@o2ib t0 exp (null) cur 1349596807 last 0
Oct  7 01:00:07 nbp1-oss8 kernel: Lustre: 65336:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 1 previous similar message
Oct  7 01:00:07 nbp1-oss8 kernel: Lustre: 65643:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 27 previous similar messages
Oct  7 01:00:15 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.14.192@o2ib [old ver: 12, new ver: 12]
Oct  7 01:00:16 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 01:00:16 nbp1-oss8 kernel: Lustre: 65341:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 6ad24dc2-e2e5-f040-3bc9-c947a10f09a3@10.151.14.192@o2ib t0 exp (null) cur 1349596816 last 0
Oct  7 01:00:16 nbp1-oss8 kernel: Lustre: 65341:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 44 previous similar messages
Oct  7 01:00:16 nbp1-oss8 kernel: Lustre: 65740:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 1941ffff-ec5b-4b81-2a64-e5a6d188bdd3@10.151.14.189@o2ib t0 exp (null) cur 1349596816 last 0
Oct  7 01:00:16 nbp1-oss8 kernel: Lustre: 65740:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 104 previous similar messages
Oct  7 01:00:25 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.14.194@o2ib [old ver: 12, new ver: 12]
Oct  7 01:00:25 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 12 previous similar messages
Oct  7 01:00:25 nbp1-oss8 kernel: Lustre: 65643:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 8bb5bb37-1775-7efa-1675-73ccc7a79207@10.151.14.194@o2ib t0 exp (null) cur 1349596825 last 0
Oct  7 01:00:25 nbp1-oss8 kernel: Lustre: 65643:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 59 previous similar messages
Oct  7 01:00:29 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.14.180@o2ib [old ver: 12, new ver: 12]
Oct  7 01:00:29 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 7 previous similar messages
Oct  7 01:00:29 nbp1-oss8 kernel: Lustre: 65993:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 88d24903-2e6a-03d2-974f-b03efb9fb9f6@10.151.14.180@o2ib t0 exp (null) cur 1349596829 last 0
Oct  7 01:00:29 nbp1-oss8 kernel: Lustre: 65336:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 88d24903-2e6a-03d2-974f-b03efb9fb9f6@10.151.14.180@o2ib t0 exp (null) cur 1349596829 last 0
Oct  7 01:00:29 nbp1-oss8 kernel: Lustre: 65336:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 90 previous similar messages
Oct  7 01:00:30 nbp1-oss8 kernel: Lustre: 65993:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 13 previous similar messages
Oct  7 01:01:08 nbp1-oss8 kernel: Lustre: nbp1-OST000f: haven't heard from client 6d2f8882-eb52-5d96-7b3c-52b09d6a8b22 (at 10.151.16.193@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061c223800, cur 1349596868 expire 1349596718 last 1349596641
Oct  7 01:01:08 nbp1-oss8 kernel: Lustre: Skipped 252 previous similar messages
Oct  7 01:01:08 nbp1-oss8 kernel: Lustre: nbp1-OST000f: haven't heard from client fb36c9cb-931e-6aa1-ba22-2d164c14bcb1 (at 10.151.8.17@o2ib) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8802f6601800, cur 1349596868 expire 1349596718 last 1349596642
Oct  7 01:01:09 nbp1-oss8 kernel: Lustre: nbp1-OST0067: haven't heard from client fb36c9cb-931e-6aa1-ba22-2d164c14bcb1 (at 10.151.8.17@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801c42fac00, cur 1349596869 expire 1349596719 last 1349596642
Oct  7 01:01:09 nbp1-oss8 kernel: Lustre: Skipped 23 previous similar messages
Oct  7 01:28:50 nbp1-oss8 kernel: Lustre: 70399:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 01:58:34 nbp1-oss8 pcp-pmie[4620]: High 1-minute load average 199load@nbp1-oss8
Oct  7 02:38:29 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.31.202@o2ib [old ver: 12, new ver: 12]
Oct  7 02:38:29 nbp1-oss8 kernel: Lustre: 65413:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 060028af-f820-06e7-718a-5f6d9f5222a4@10.151.31.202@o2ib t0 exp (null) cur 1349602709 last 0
Oct  7 02:38:29 nbp1-oss8 kernel: Lustre: 65413:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 11 previous similar messages
Oct  7 02:39:10 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.31.201@o2ib [old ver: 12, new ver: 12]
Oct  7 02:39:10 nbp1-oss8 kernel: Lustre: 65328:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 095098aa-6171-edf4-7055-8f04e270f053@10.151.31.201@o2ib t0 exp (null) cur 1349602750 last 0
Oct  7 02:39:11 nbp1-oss8 kernel: Lustre: 65328:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 14 previous similar messages
Oct  7 02:39:20 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client fb177c66-4368-544c-0322-7a0ad8d334f7 (at 10.151.31.202@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88077c238800, cur 1349602760 expire 1349602610 last 1349602533
Oct  7 02:39:20 nbp1-oss8 kernel: Lustre: Skipped 349 previous similar messages
Oct  7 02:39:21 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.31.204@o2ib [old ver: 12, new ver: 12]
Oct  7 02:39:21 nbp1-oss8 kernel: Lustre: 65714:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from b6a72a82-ab47-bbc8-ded9-c5b32a082915@10.151.31.204@o2ib t0 exp (null) cur 1349602761 last 0
Oct  7 02:39:21 nbp1-oss8 kernel: Lustre: 65714:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 27 previous similar messages
Oct  7 03:17:15 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client f0c03531-8777-198c-9c63-0598218dbae9 (at 10.151.30.247@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880608b06000, cur 1349605035 expire 1349604885 last 1349604808
Oct  7 03:17:15 nbp1-oss8 kernel: Lustre: Skipped 59 previous similar messages
Oct  7 03:17:15 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client f0c03531-8777-198c-9c63-0598218dbae9 (at 10.151.30.247@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88064218fc00, cur 1349605035 expire 1349604885 last 1349604808
Oct  7 03:19:56 nbp1-oss8 kernel: Lustre: 65340:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 8f023035-9a39-9d83-8c63-3057faa723eb@10.151.30.247@o2ib t0 exp (null) cur 1349605196 last 0
Oct  7 03:19:56 nbp1-oss8 kernel: Lustre: 65340:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 12 previous similar messages
Oct  7 03:40:17 nbp1-oss8 kernel: Lustre: nbp1-OST0067: haven't heard from client 09f8afd4-5902-47a3-e886-dcce889f8560 (at 10.151.56.131@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810a7b6b000, cur 1349606417 expire 1349606267 last 1349606190
Oct  7 03:40:17 nbp1-oss8 kernel: Lustre: Skipped 13 previous similar messages
Oct  7 03:40:17 nbp1-oss8 kernel: Lustre: nbp1-OST005f: haven't heard from client 09f8afd4-5902-47a3-e886-dcce889f8560 (at 10.151.56.131@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880528954400, cur 1349606417 expire 1349606267 last 1349606190
Oct  7 03:41:03 nbp1-oss8 kernel: Lustre: 73334:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 04:42:34 nbp1-oss8 pcp-pmie[4620]: High 1-minute load average 242load@nbp1-oss8
Oct  7 04:44:00 nbp1-oss8 kernel: Lustre: 65667:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0017: connection from 8005ec8c-2dee-0b7a-9fc8-435e97b53fdb@10.151.56.131@o2ib t0 exp (null) cur 1349610240 last 0
Oct  7 04:44:00 nbp1-oss8 kernel: Lustre: 65667:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 12 previous similar messages
Oct  7 05:03:12 nbp1-oss8 kernel: Lustre: 73035:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 05:30:54 nbp1-oss8 kernel: Lustre: 70393:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 05:35:27 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.33.64@o2ib [old ver: 12, new ver: 12]
Oct  7 05:35:27 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 05:35:27 nbp1-oss8 kernel: Lustre: 65834:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 7f8da12c-b747-d1c3-d190-8e3a850c2073@10.151.33.64@o2ib t0 exp (null) cur 1349613327 last 0
Oct  7 05:35:27 nbp1-oss8 kernel: Lustre: 65834:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 13 previous similar messages
Oct  7 05:36:15 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client 3ce571fe-3cda-9c85-2884-fe2716ec8ad5 (at 10.151.33.64@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881002763400, cur 1349613375 expire 1349613225 last 1349613148
Oct  7 05:36:15 nbp1-oss8 kernel: Lustre: Skipped 13 previous similar messages
Oct  7 05:36:15 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 3ce571fe-3cda-9c85-2884-fe2716ec8ad5 (at 10.151.33.64@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880db99dc800, cur 1349613375 expire 1349613225 last 1349613148
Oct  7 05:39:22 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.33.66@o2ib [old ver: 12, new ver: 12]
Oct  7 05:39:22 nbp1-oss8 kernel: Lustre: 65683:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from ce60545e-05ce-e744-6c4b-17bf8a4a641f@10.151.33.66@o2ib t0 exp (null) cur 1349613562 last 0
Oct  7 05:39:22 nbp1-oss8 kernel: Lustre: 65359:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from ce60545e-05ce-e744-6c4b-17bf8a4a641f@10.151.33.66@o2ib t0 exp (null) cur 1349613562 last 0
Oct  7 05:39:22 nbp1-oss8 kernel: Lustre: 65683:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 12 previous similar messages
Oct  7 05:40:23 nbp1-oss8 kernel: Lustre: nbp1-OST0017: haven't heard from client a10ac40b-c398-d886-ab68-cac9b6eceafd (at 10.151.33.66@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0e7fdc00, cur 1349613623 expire 1349613473 last 1349613396
Oct  7 05:40:23 nbp1-oss8 kernel: Lustre: Skipped 13 previous similar messages
Oct  7 05:41:14 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.33.61@o2ib [old ver: 12, new ver: 12]
Oct  7 05:41:14 nbp1-oss8 kernel: Lustre: 65420:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 7f064a56-c8ec-1057-c3f3-615abc4abdd6@10.151.33.61@o2ib t0 exp (null) cur 1349613674 last 0
Oct  7 05:41:14 nbp1-oss8 kernel: Lustre: 65420:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 14 previous similar messages
Oct  7 05:41:39 nbp1-oss8 kernel: Lustre: nbp1-OST0027: haven't heard from client 8d7cdd95-428f-008a-6acb-e28a2115ba78 (at 10.151.33.61@o2ib) in 216 seconds. I think it's dead, and I am evicting it. exp ffff880678db6c00, cur 1349613699 expire 1349613549 last 1349613483
Oct  7 05:41:39 nbp1-oss8 kernel: Lustre: Skipped 14 previous similar messages
Oct  7 06:13:49 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.55.187@o2ib [old ver: 12, new ver: 12]
Oct  7 06:13:49 nbp1-oss8 kernel: Lustre: 65794:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0017: connection from 2300d352-b149-4d15-3664-335b75560c00@10.151.55.187@o2ib t0 exp (null) cur 1349615629 last 0
Oct  7 06:13:49 nbp1-oss8 kernel: Lustre: 65794:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 25 previous similar messages
Oct  7 06:13:57 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.56.77@o2ib [old ver: 12, new ver: 12]
Oct  7 06:13:57 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 06:13:57 nbp1-oss8 kernel: Lustre: 65666:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 6544d6b1-3817-14dd-9e5e-be463ee307da@10.151.55.89@o2ib t0 exp (null) cur 1349615637 last 0
Oct  7 06:13:57 nbp1-oss8 kernel: Lustre: 65666:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 114 previous similar messages
Oct  7 06:13:58 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.41.26@o2ib [old ver: 12, new ver: 12]
Oct  7 06:13:58 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 9 previous similar messages
Oct  7 06:13:58 nbp1-oss8 kernel: Lustre: 65991:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 745d7648-2df7-73f6-aef1-5bc965b3a4cf@10.151.41.26@o2ib t0 exp (null) cur 1349615638 last 0
Oct  7 06:13:58 nbp1-oss8 kernel: Lustre: 65991:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 38 previous similar messages
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.55.62@o2ib [old ver: 12, new ver: 12]
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 9 previous similar messages
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65324:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0017: connection from 9eefba36-fbde-1f91-04da-8aceec27382e@10.151.55.62@o2ib t0 exp (null) cur 1349615639 last 0
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65692:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 9eefba36-fbde-1f91-04da-8aceec27382e@10.151.55.62@o2ib t0 exp (null) cur 1349615639 last 0
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65944:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 9eefba36-fbde-1f91-04da-8aceec27382e@10.151.55.62@o2ib t0 exp (null) cur 1349615639 last 0
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65692:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 122 previous similar messages
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65944:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 122 previous similar messages
Oct  7 06:13:59 nbp1-oss8 kernel: Lustre: 65324:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 11 previous similar messages
Oct  7 06:14:12 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.40.39@o2ib [old ver: 12, new ver: 12]
Oct  7 06:14:12 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 5 previous similar messages
Oct  7 06:14:12 nbp1-oss8 kernel: Lustre: 66001:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from b71a8501-6568-393c-73f2-dcfc43c2d5cc@10.151.40.39@o2ib t0 exp (null) cur 1349615652 last 0
Oct  7 06:14:12 nbp1-oss8 kernel: Lustre: 66001:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 85 previous similar messages
Oct  7 06:14:16 nbp1-oss8 kernel: Lustre: 1903:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.56.15@o2ib [old ver: 12, new ver: 12]
Oct  7 06:14:16 nbp1-oss8 kernel: Lustre: 1903:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 4 previous similar messages
Oct  7 06:14:41 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client 506380d8-3ce5-5ccc-9b60-2112b671c7e1 (at 10.151.55.54@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88e5d000, cur 1349615681 expire 1349615531 last 1349615454
Oct  7 06:14:41 nbp1-oss8 kernel: Lustre: Skipped 14 previous similar messages
Oct  7 06:38:54 nbp1-oss8 kernel: Lustre: 73012:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.31.9@o2ib [old ver: 12, new ver: 12]
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 8 previous similar messages
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 65842:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from 4119c168-8562-0676-a8b5-bee36e4820bd@10.151.31.9@o2ib t0 exp (null) cur 1349617867 last 0
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 65896:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 4119c168-8562-0676-a8b5-bee36e4820bd@10.151.31.9@o2ib t0 exp (null) cur 1349617867 last 0
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 65896:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 183 previous similar messages
Oct  7 06:51:07 nbp1-oss8 kernel: Lustre: 65842:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 10 previous similar messages
Oct  7 06:52:32 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client 26e84849-3e29-7844-31d4-4a4e0b7dcd3b (at 10.151.31.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809cf206400, cur 1349617952 expire 1349617802 last 1349617725
Oct  7 06:52:32 nbp1-oss8 kernel: Lustre: Skipped 629 previous similar messages
Oct  7 06:52:32 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 26e84849-3e29-7844-31d4-4a4e0b7dcd3b (at 10.151.31.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881099e34000, cur 1349617952 expire 1349617802 last 1349617725
Oct  7 07:05:01 nbp1-oss8 auditd[3328]: Audit daemon rotating log files
Oct  7 07:28:02 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.15.241@o2ib [old ver: 12, new ver: 12]
Oct  7 07:28:02 nbp1-oss8 kernel: Lustre: 65335:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0017: connection from 1e74b67a-4ced-00fa-1700-c78b774b6ad5@10.151.15.241@o2ib t0 exp (null) cur 1349620082 last 0
Oct  7 07:28:02 nbp1-oss8 kernel: Lustre: 65335:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 11 previous similar messages
Oct  7 07:29:54 nbp1-oss8 kernel: Lustre: nbp1-OST0047: haven't heard from client 3533f6f7-1f1e-6997-8bea-167133f30705 (at 10.151.15.241@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881020f79c00, cur 1349620194 expire 1349620044 last 1349619967
Oct  7 07:29:54 nbp1-oss8 kernel: Lustre: Skipped 13 previous similar messages
Oct  7 07:29:54 nbp1-oss8 kernel: Lustre: nbp1-OST003f: haven't heard from client 3533f6f7-1f1e-6997-8bea-167133f30705 (at 10.151.15.241@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810398b9400, cur 1349620194 expire 1349620044 last 1349619967
Oct  7 08:18:02 nbp1-oss8 syslog-ng[3346]: Log statistics; processed='destination(d_messages)=34006', processed='destination(d_syslog)=467266', processed='src.internal(src#0)=243', stamp='src.internal(src#0)=1349594282', processed='center(queued)=0', processed='destination(acpid)=0', processed='destination(sec-logger)=0', processed='destination(uselog)=0', processed='destination(firewall)=0', dropped='dst.udp(sec-logger#0,10.151.0.15:514)=0', processed='dst.udp(sec-logger#0,10.151.0.15:514)=0', stored='dst.udp(sec-logger#0,10.151.0.15:514)=0', processed='global(payload_reallocs)=0', processed='destination(mail)=159', processed='destination(netmgm)=0', processed='destination(console)=30925', processed='destination(cron)=139518', processed='global(msg_clones)=0', processed='destination(d_warn)=31088', processed='global(sdata_updates)=0', processed='destination(localmessages)=33', processed='center(received)=0', processed='destination(loghost)=466771', processed='source(src)=467425', dropped='dst.udp(loghost#0,admin:514)=0', processed='dst.udp(loghost#0,admin:514)=466771', stored='dst.udp(loghost#0,admin:514)=0'
Oct  7 08:46:29 nbp1-oss8 kernel: Lustre: 73229:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 08:47:19 nbp1-oss8 kernel: Lustre: 73351:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 08:48:07 nbp1-oss8 kernel: Lustre: 65518:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 09:46:16 nbp1-oss8 kernel: Lustre: 73453:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 10:05:19 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 79bdeb8b-a873-bb8b-487c-9590e776ab37 (at 10.151.39.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a41807400, cur 1349629519 expire 1349629369 last 1349629292
Oct  7 10:05:19 nbp1-oss8 kernel: Lustre: Skipped 13 previous similar messages
Oct  7 10:05:19 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 0ee74238-794b-abdf-bf72-1f600153b5aa (at 10.151.38.168@o2ib) in 216 seconds. I think it's dead, and I am evicting it. exp ffff8802e1bbc400, cur 1349629519 expire 1349629369 last 1349629303
Oct  7 10:23:20 nbp1-oss8 kernel: Lustre: 70464:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 10:39:36 nbp1-oss8 kernel: Lustre: 65721:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from ff27631d-34fd-2c98-5fa6-613feaffc69c@10.151.39.111@o2ib t0 exp (null) cur 1349631576 last 0
Oct  7 10:39:36 nbp1-oss8 kernel: Lustre: 65721:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 13 previous similar messages
Oct  7 10:40:36 nbp1-oss8 kernel: Lustre: 65314:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from e3d9db10-95bb-0971-57ea-be7fabcaa55e@10.151.38.168@o2ib t0 exp (null) cur 1349631636 last 0
Oct  7 10:40:36 nbp1-oss8 kernel: Lustre: 65314:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 12 previous similar messages
Oct  7 11:01:52 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 249s: evicting client at 10.151.42.128@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8803dcd90d80/0x51d65865e83a14b9 lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 2816 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7d16e40b5c4be50 expref: 9 pid: 65379 timeout 10254553252
Oct  7 11:01:52 nbp1-oss8 kernel: LustreError: 65293:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88067a3bac00 x1408944816713591/t0(0) o105->nbp1-OST006f@10.151.42.128@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Oct  7 11:01:52 nbp1-oss8 kernel: LustreError: 65293:0:(ldlm_lockd.c:654:ldlm_handle_ast_error()) ### client (nid 10.151.42.128@o2ib) returned 0 from completion AST ns: filter-nbp1-OST006f_UUID lock: ffff8806f510b6c0/0x51d65865e83a14c0 lrc: 4/0,0 mode: PW/PW res: 49291430/0 rrc: 2817 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7d16e40b5c4be57 expref: 9 pid: 65379 timeout 10254779108
Oct  7 11:01:53 nbp1-oss8 kernel: LustreError: 65293:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88067a3bac00 x1408944816713592/t0(0) o105->nbp1-OST006f@10.151.42.128@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Oct  7 11:01:53 nbp1-oss8 kernel: LustreError: 65293:0:(ldlm_lockd.c:654:ldlm_handle_ast_error()) ### client (nid 10.151.42.128@o2ib) returned 0 from completion AST ns: filter-nbp1-OST006f_UUID lock: ffff8802c630a900/0x51d65865e83a14c7 lrc: 4/0,0 mode: PW/PW res: 49291430/0 rrc: 2816 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7d16e40b5c4be5e expref: 8 pid: 65379 timeout 10254779273
Oct  7 11:02:12 nbp1-oss8 kernel: Lustre: 65995:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from f933e5fd-137d-f1ef-f610-677efa4d5c0e@10.151.42.128@o2ib t0 exp (null) cur 1349632932 last 0
Oct  7 11:05:38 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 475s: evicting client at 10.151.42.131@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff88048c6ae900/0x51d65865e83a14ce lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 2810 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x9252f7da37d55ecc expref: 8 pid: 65379 timeout 10254779439
Oct  7 11:05:55 nbp1-oss8 kernel: Lustre: 65771:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from 1e4c1080-1b97-e7a6-1562-b29f3884142a@10.151.42.131@o2ib t0 exp (null) cur 1349633155 last 0
Oct  7 11:09:08 nbp1-oss8 kernel: Lustre: 72978:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 11:09:24 nbp1-oss8 kernel: LustreError: 14:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 701s: evicting client at 10.151.42.133@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff880761b996c0/0x51d65865e83a14d5 lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 2805 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x49a3403282790c65 expref: 9 pid: 65379 timeout 10255005108
Oct  7 11:09:25 nbp1-oss8 kernel: Lustre: 65379:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from bd840a03-c5ae-affc-5fb7-9078eba0dd39@10.151.42.133@o2ib t0 exp (null) cur 1349633365 last 0
Oct  7 11:13:10 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 927s: evicting client at 10.151.42.138@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8801e7cadb40/0x51d65865e83a14dc lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 2799 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xd97c377a3cbbed98 expref: 8 pid: 65379 timeout 10255231109
Oct  7 11:13:13 nbp1-oss8 kernel: Lustre: 73026:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 11:13:23 nbp1-oss8 kernel: Lustre: 65893:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from 8f977fdb-bcc0-f044-5c2c-87c77b8ca890@10.151.42.138@o2ib t0 exp (null) cur 1349633603 last 0
Oct  7 11:13:45 nbp1-oss8 kernel: Lustre: 65518:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 11:16:56 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 1153s: evicting client at 10.151.42.136@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8807a0b19240/0x51d65865e83a14e3 lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 2794 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xa2a996d11836560a expref: 9 pid: 65379 timeout 10255457109
Oct  7 11:17:11 nbp1-oss8 kernel: Lustre: 65683:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from ba0234ea-272f-b588-e547-d1e6c2ae4953@10.151.42.136@o2ib t0 exp (null) cur 1349633831 last 0
Oct  7 11:20:42 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 1379s: evicting client at 10.151.42.137@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8801a10326c0/0x51d65865e83a14ea lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 461 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x1081f56f25f54d51 expref: 5 pid: 65379 timeout 10255683108
Oct  7 11:20:54 nbp1-oss8 kernel: Lustre: 65638:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from ac2b0cef-2d54-364b-18ec-8ceca1c7b9cc@10.151.42.137@o2ib t0 exp (null) cur 1349634054 last 0
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.17.56@o2ib [old ver: 12, new ver: 12]
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 65936:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634143/real 1349634233]  req@ffff8804d0649000 x1408944816871383/t0(0) o106->nbp1-OST000f@10.151.17.56@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634485 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 65936:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 653 previous similar messages
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.18.23@o2ib [old ver: 12, new ver: 12]
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 65303:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 0d6e512a-d9a3-e555-cae3-5332f75dd726@10.151.17.56@o2ib t0 exp (null) cur 1349634233 last 0
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST000f: A client on nid 10.151.17.56@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: Skipped 14 previous similar messages
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.13.216@o2ib rejected: o2iblnd fatal error
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 109 previous similar messages
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 65954:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634233/real 1349634233]  req@ffff88069a8ab800 x1408944816871485/t0(0) o106->nbp1-OST0057@10.151.13.216@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634575 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 65954:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST0057: A client on nid 10.151.13.216@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 1903:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.13.223@o2ib
Oct  7 11:23:53 nbp1-oss8 kernel: Lustre: 1903:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 118 previous similar messages
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.3.66@o2ib rejected: o2iblnd fatal error
Oct  7 11:23:53 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 38 previous similar messages
Oct  7 11:23:54 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST002f: A client on nid 10.151.13.223@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:23:54 nbp1-oss8 kernel: LustreError: Skipped 7 previous similar messages
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.3.59@o2ib [old ver: 12, new ver: 12]
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 4 previous similar messages
Oct  7 11:23:55 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST006f: A client on nid 10.151.13.216@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 65735:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634143/real 1349634235]  req@ffff8803975a7400 x1408944816871497/t0(0) o106->nbp1-OST004f@10.151.3.102@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634485 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.3.102@o2ib
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 43 previous similar messages
Oct  7 11:23:55 nbp1-oss8 kernel: LustreError: 1903:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.3.102@o2ib rejected: o2iblnd fatal error
Oct  7 11:23:55 nbp1-oss8 kernel: LustreError: 1903:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 9 previous similar messages
Oct  7 11:23:55 nbp1-oss8 kernel: Lustre: 65735:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 96 previous similar messages
Oct  7 11:23:56 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.3.106@o2ib
Oct  7 11:23:56 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 215 previous similar messages
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.6.119@o2ib [old ver: 12, new ver: 12]
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 2 previous similar messages
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 65639:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634143/real 1349634238]  req@ffff880d1b0b4800 x1408944816871528/t0(0) o106->nbp1-OST000f@10.151.6.119@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634485 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 65639:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 431 previous similar messages
Oct  7 11:23:58 nbp1-oss8 kernel: LustreError: 1905:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.6.119@o2ib rejected: o2iblnd fatal error
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 1893:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.6.119@o2ib
Oct  7 11:23:58 nbp1-oss8 kernel: Lustre: 1893:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 107 previous similar messages
Oct  7 11:23:58 nbp1-oss8 kernel: LustreError: 1905:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 236 previous similar messages
Oct  7 11:23:58 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST000f: A client on nid 10.151.6.119@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:23:58 nbp1-oss8 kernel: LustreError: Skipped 5 previous similar messages
Oct  7 11:24:00 nbp1-oss8 kernel: Lustre: 65811:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634240/real 1349634240]  req@ffff88104b875800 x1408944816871522/t0(0) o106->nbp1-OST003f@10.151.6.103@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634582 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Oct  7 11:24:00 nbp1-oss8 kernel: Lustre: 65811:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 525 previous similar messages
Oct  7 11:24:00 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.6.103@o2ib rejected: o2iblnd fatal error
Oct  7 11:24:00 nbp1-oss8 kernel: Lustre: 1902:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.6.103@o2ib
Oct  7 11:24:00 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 293 previous similar messages
Oct  7 11:24:00 nbp1-oss8 kernel: Lustre: 1902:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 318 previous similar messages
Oct  7 11:24:03 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.6.120@o2ib [old ver: 12, new ver: 12]
Oct  7 11:24:03 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 5 previous similar messages
Oct  7 11:24:04 nbp1-oss8 kernel: Lustre: 65924:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634244/real 1349634244]  req@ffff880df411fc00 x1408944816871421/t0(0) o106->nbp1-OST0047@10.151.6.120@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634586 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Oct  7 11:24:04 nbp1-oss8 kernel: Lustre: 65924:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 546 previous similar messages
Oct  7 11:24:04 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) 10.151.6.120@o2ib rejected: o2iblnd fatal error
Oct  7 11:24:04 nbp1-oss8 kernel: LustreError: 1891:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 268 previous similar messages
Oct  7 11:24:04 nbp1-oss8 kernel: Lustre: 1888:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Conn race 10.151.6.120@o2ib
Oct  7 11:24:04 nbp1-oss8 kernel: Lustre: 1888:0:(o2iblnd_cb.c:2340:kiblnd_passive_connect()) Skipped 208 previous similar messages
Oct  7 11:24:04 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST0047: A client on nid 10.151.6.120@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:24:04 nbp1-oss8 kernel: LustreError: Skipped 12 previous similar messages
Oct  7 11:24:11 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.3.99@o2ib [old ver: 12, new ver: 12]
Oct  7 11:24:11 nbp1-oss8 kernel: Lustre: 65653:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 6f63e289-5e13-f47f-70ee-409dccdf7943@10.151.3.99@o2ib t0 exp (null) cur 1349634251 last 0
Oct  7 11:24:11 nbp1-oss8 kernel: Lustre: 65653:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 234 previous similar messages
Oct  7 11:24:16 nbp1-oss8 kernel: Lustre: 65893:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1349634143/real 1349634256]  req@ffff880b9cc21c00 x1408944816871393/t0(0) o106->nbp1-OST001f@10.151.3.104@o2ib:15/16 lens 296/232 e 0 to 1 dl 1349634485 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Oct  7 11:24:16 nbp1-oss8 kernel: Lustre: 65893:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 42 previous similar messages
Oct  7 11:24:16 nbp1-oss8 kernel: LustreError: 138-a: nbp1-OST001f: A client on nid 10.151.3.104@o2ib was evicted due to a lock glimpse callback time out: rc -107
Oct  7 11:24:16 nbp1-oss8 kernel: LustreError: Skipped 1 previous similar message
Oct  7 11:24:28 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 1605s: evicting client at 10.151.45.66@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8806d1f32d80/0x51d65865e83a19ac lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 460 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7066f20a27bbcfc7 expref: 9 pid: 65661 timeout 10255909109
Oct  7 11:24:46 nbp1-oss8 kernel: Lustre: 65726:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from 29810dea-7aef-2c9d-6ce8-455a9b020280@10.151.45.66@o2ib t0 exp (null) cur 1349634285 last 0
Oct  7 11:24:46 nbp1-oss8 kernel: Lustre: 65726:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 13 previous similar messages
Oct  7 11:25:30 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 51c86b51-3c21-4ea6-2c2b-507613fb08c7 (at 10.151.3.99@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff6c39c00, cur 1349634330 expire 1349634180 last 1349634103
Oct  7 11:25:30 nbp1-oss8 kernel: Lustre: Skipped 28 previous similar messages
Oct  7 11:25:30 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 26a0f7f6-60c1-9333-2e6d-1deac94ebd21 (at 10.151.3.102@o2ib) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8811e793e400, cur 1349634330 expire 1349634180 last 1349634104
Oct  7 11:25:40 nbp1-oss8 kernel: Lustre: nbp1-OST000f: haven't heard from client 6ebf4eef-00a4-0d15-fc3b-64335b022912 (at 10.151.6.103@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806ccb2b800, cur 1349634340 expire 1349634190 last 1349634113
Oct  7 11:25:40 nbp1-oss8 kernel: Lustre: Skipped 15 previous similar messages
Oct  7 11:28:14 nbp1-oss8 kernel: LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 1831s: evicting client at 10.151.48.59@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff88061e78bb40/0x51d65865e83a1a00 lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 454 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x13ee38d7eda9f7cc expref: 9 pid: 65661 timeout 10256135108
Oct  7 11:28:31 nbp1-oss8 kernel: Lustre: 65388:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from 88ae0e4a-3177-1d5d-5e51-5b115631e9b3@10.151.48.59@o2ib t0 exp (null) cur 1349634511 last 0
Oct  7 11:28:45 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.17.97@o2ib [old ver: 12, new ver: 12]
Oct  7 11:28:45 nbp1-oss8 kernel: LustreError: 9333:0:(events.c:382:server_bulk_callback()) event type 4, status -103, desc ffff880710c46140
Oct  7 11:28:45 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 11:29:56 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client 765aa731-a68b-c8a6-7ae1-dbbf9eb25631 (at 10.151.17.98@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ed617bc00, cur 1349634596 expire 1349634446 last 1349634369
Oct  7 11:29:56 nbp1-oss8 kernel: Lustre: Skipped 223 previous similar messages
Oct  7 11:29:56 nbp1-oss8 kernel: LustreError: 70421:0:(ldlm_lib.c:2631:target_bulk_io()) @@@ Eviction on bulk PUT  req@ffff8806e66e7000 x1410348654224615/t0(0) o3->ac65cccf-26a9-2c16-ce05-a2bec5dd5307@10.151.17.97@o2ib:0/0 lens 456/400 e 1 to 0 dl 1349634792 ref 1 fl Interpret:/0/0 rc 0/0
Oct  7 11:29:56 nbp1-oss8 kernel: Lustre: nbp1-OST0077: Bulk IO read error with ac65cccf-26a9-2c16-ce05-a2bec5dd5307 (at 10.151.17.97@o2ib), client will retry: rc -107
Oct  7 11:29:56 nbp1-oss8 kernel: Lustre: Skipped 1 previous similar message
Oct  7 11:29:56 nbp1-oss8 kernel: LustreError: 70421:0:(ldlm_lib.c:2631:target_bulk_io()) Skipped 1 previous similar message
Oct  7 11:29:57 nbp1-oss8 kernel: Lustre: nbp1-OST0077: Bulk IO read error with ac65cccf-26a9-2c16-ce05-a2bec5dd5307 (at 10.151.17.97@o2ib), client will retry: rc -107
Oct  7 11:29:59 nbp1-oss8 kernel: Lustre: nbp1-OST0007: haven't heard from client 4ee7be5b-9c4f-eb75-137c-cb0bf4abaa1c (at 10.151.7.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6b3c4400, cur 1349634599 expire 1349634449 last 1349634372
Oct  7 11:29:59 nbp1-oss8 kernel: Lustre: Skipped 20 previous similar messages
Oct  7 11:32:00 nbp1-oss8 kernel: LustreError: 70379:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 2057s: evicting client at 10.151.48.66@o2ib  ns: filter-nbp1-OST006f_UUID lock: ffff8803b1559900/0x51d65865e83a1a1c lrc: 3/0,0 mode: PW/PW res: 49291430/0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x31356edf6f3a5c31 expref: 4 pid: 65661 timeout 10256361109
Oct  7 11:32:19 nbp1-oss8 kernel: Lustre: 65945:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST006f: connection from c8c18ad3-4040-702c-f1e2-4c8db2348cc1@10.151.48.66@o2ib t0 exp (null) cur 1349634739 last 0
Oct  7 11:32:19 nbp1-oss8 kernel: Lustre: 65945:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 301 previous similar messages
Oct  7 11:47:47 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.7.168@o2ib [old ver: 12, new ver: 12]
Oct  7 11:47:48 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 20 previous similar messages
Oct  7 11:47:48 nbp1-oss8 kernel: Lustre: 65971:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0017: connection from a1a207fe-73b8-4c1c-1932-433f369bf060@10.151.7.168@o2ib t0 exp (null) cur 1349635668 last 0
Oct  7 11:47:48 nbp1-oss8 kernel: Lustre: 65988:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST000f: connection from a1a207fe-73b8-4c1c-1932-433f369bf060@10.151.7.168@o2ib t0 exp (null) cur 1349635668 last 0
Oct  7 11:47:48 nbp1-oss8 kernel: Lustre: 65988:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 4 previous similar messages
Oct  7 11:47:48 nbp1-oss8 kernel: Lustre: 65971:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 37 previous similar messages
Oct  7 11:47:50 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.4.188@o2ib [old ver: 12, new ver: 12]
Oct  7 11:47:50 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 14 previous similar messages
Oct  7 11:47:53 nbp1-oss8 kernel: LustreError: 9333:0:(events.c:382:server_bulk_callback()) event type 4, status -103, desc ffff880c8271cb40
Oct  7 11:47:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.9.176@o2ib [old ver: 12, new ver: 12]
Oct  7 11:47:55 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 21 previous similar messages
Oct  7 11:48:15 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.9.197@o2ib [old ver: 12, new ver: 12]
Oct  7 11:48:15 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 7 previous similar messages
Oct  7 11:48:53 nbp1-oss8 kernel: Lustre: nbp1-OST0077: haven't heard from client d4faca93-0b57-285a-5ed7-c67650522ba5 (at 10.151.14.73@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cab245c00, cur 1349635733 expire 1349635583 last 1349635506
Oct  7 11:48:53 nbp1-oss8 kernel: Lustre: Skipped 293 previous similar messages
Oct  7 11:48:53 nbp1-oss8 kernel: LustreError: 73268:0:(ldlm_lib.c:2631:target_bulk_io()) @@@ Eviction on bulk PUT  req@ffff88068c7b6000 x1409711161923013/t0(0) o3->c800ffa2-cd65-8bf2-c830-262e35652fa4@10.151.7.117@o2ib:0/0 lens 456/400 e 1 to 0 dl 1349635932 ref 1 fl Interpret:/0/0 rc 0/0
Oct  7 11:48:53 nbp1-oss8 kernel: Lustre: nbp1-OST0077: Bulk IO read error with c800ffa2-cd65-8bf2-c830-262e35652fa4 (at 10.151.7.117@o2ib), client will retry: rc -107
Oct  7 11:48:58 nbp1-oss8 kernel: Lustre: nbp1-OST006f: haven't heard from client e333cd92-44cc-4426-5716-454224599845 (at 10.151.7.168@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881070b42800, cur 1349635738 expire 1349635588 last 1349635511
Oct  7 11:48:58 nbp1-oss8 kernel: Lustre: Skipped 45 previous similar messages
Oct  7 11:49:02 nbp1-oss8 kernel: Lustre: nbp1-OST0057: haven't heard from client aefdc326-8317-af09-6f81-afea6c6b785e (at 10.151.6.27@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810a8325000, cur 1349635742 expire 1349635592 last 1349635515
Oct  7 11:49:02 nbp1-oss8 kernel: Lustre: Skipped 45 previous similar messages
Oct  7 12:46:49 nbp1-oss8 kernel: Lustre: 73285:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 13:03:55 nbp1-oss8 kernel: Lustre: 70442:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 14:17:11 nbp1-oss8 kernel: Lustre: 73213:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 14:25:39 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.13.32@o2ib [old ver: 12, new ver: 12]
Oct  7 14:25:39 nbp1-oss8 kernel: Lustre: 65997:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 855a750e-b7a9-8678-4005-bd8b1029d757@10.151.13.32@o2ib t0 exp (null) cur 1349645139 last 0
Oct  7 14:25:39 nbp1-oss8 kernel: Lustre: 65997:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 619 previous similar messages
Oct  7 14:27:25 nbp1-oss8 kernel: Lustre: nbp1-OST000f: haven't heard from client 40a964d3-a53a-4771-578a-9deef3bbef55 (at 10.151.13.31@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88027dcd2c00, cur 1349645245 expire 1349645095 last 1349645018
Oct  7 14:27:25 nbp1-oss8 kernel: Lustre: Skipped 597 previous similar messages
Oct  7 14:32:54 nbp1-oss8 kernel: Lustre: 73290:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 14:46:38 nbp1-oss8 kernel: Lustre: 73026:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 14:59:13 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.15.156@o2ib [old ver: 12, new ver: 12]
Oct  7 14:59:13 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Skipped 1 previous similar message
Oct  7 14:59:13 nbp1-oss8 kernel: Lustre: 65732:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 536dbc11-b8ed-e73f-c556-bf02c24179ab@10.151.15.156@o2ib t0 exp (null) cur 1349647153 last 0
Oct  7 14:59:13 nbp1-oss8 kernel: Lustre: 65732:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 28 previous similar messages
Oct  7 14:59:15 nbp1-oss8 kernel: Lustre: 1891:0:(o2iblnd_cb.c:2328:kiblnd_passive_connect()) Conn stale 10.151.14.115@o2ib [old ver: 12, new ver: 12]
Oct  7 14:59:15 nbp1-oss8 kernel: Lustre: 65826:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST0007: connection from 62303ad3-a445-891b-a64e-88b057457126@10.151.14.115@o2ib t0 exp (null) cur 1349647155 last 0
Oct  7 14:59:15 nbp1-oss8 kernel: Lustre: 65826:0:(ldlm_lib.c:947:target_handle_connect()) Skipped 14 previous similar messages
Oct  7 15:01:09 nbp1-oss8 kernel: Lustre: nbp1-OST005f: haven't heard from client b3b13df0-c51a-c602-5066-4113e524aeca (at 10.151.14.115@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881042c91400, cur 1349647269 expire 1349647119 last 1349647042
Oct  7 15:01:09 nbp1-oss8 kernel: Lustre: Skipped 29 previous similar messages
Oct  7 15:01:09 nbp1-oss8 kernel: Lustre: nbp1-OST005f: haven't heard from client 2a5e8a84-d5ac-90b6-6525-f0d9c8491f94 (at 10.151.15.156@o2ib) in 224 seconds. I think it's dead, and I am evicting it. exp ffff88104c034400, cur 1349647269 expire 1349647119 last 1349647045
Oct  7 15:12:11 nbp1-oss8 kernel: LustreError: 73265:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 227s: evicting client at 10.151.25.210@o2ib  ns: filter-nbp1-OST005f_UUID lock: ffff88043b7ef240/0x51d65865e8ed65af lrc: 3/0,0 mode: PR/PR res: 49954402/0 rrc: 2 type: EXT [0->18446744073709551615] (req 856064->868351) flags: 0x20 remote: 0x3bf5f5ecface519f expref: 4 pid: 65721 timeout 10269572925
Oct  7 15:12:15 nbp1-oss8 kernel: Lustre: 65383:0:(ldlm_lib.c:947:target_handle_connect()) nbp1-OST005f: connection from 39a11ee0-f37f-e2c4-26af-212707adebb2@10.151.25.210@o2ib t0 exp (null) cur 1349647935 last 0
Oct  7 15:23:19 nbp1-oss8 kernel: Lustre: 65441:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 15:23:53 nbp1-oss8 kernel: Lustre: 73003:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 15:24:33 nbp1-oss8 kernel: Lustre: 70457:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 15:24:34 nbp1-oss8 pcp-pmie[4620]: High 1-minute load average 107load@nbp1-oss8
Oct  7 15:25:09 nbp1-oss8 kernel: Lustre: 65544:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 15:31:19 nbp1-oss8 kernel: Lustre: 72985:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Oct  7 15:47:11 nbp1-oss8 kernel: Lustre: 65441:0:(quota_interface.c:532:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
