Jun 17 11:57:55 sklusp01a syslogd 1.4.1: restart. Jun 17 21:55:17 sklusp01a kernel: DLM (built Nov 16 2010 17:59:42) installed Jun 17 21:55:17 sklusp01a kernel: GFS2 (built Nov 16 2010 18:00:55) installed Jun 17 21:55:17 sklusp01a kernel: Lock_DLM (built Nov 16 2010 18:01:06) installed Jun 17 21:55:17 sklusp01a ccsd[1475]: Starting ccsd 2.0.115: Jun 17 21:55:17 sklusp01a ccsd[1475]: Built: Mar 16 2010 10:28:57 Jun 17 21:55:17 sklusp01a ccsd[1475]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Jun 17 21:55:17 sklusp01a ccsd[1475]: cluster.conf (cluster name = lusp01cl, version = 6) found. Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote copy of cluster.conf is from quorate node. Jun 17 21:55:17 sklusp01a ccsd[1475]: Local version # : 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote version #: 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote copy of cluster.conf is from quorate node. Jun 17 21:55:17 sklusp01a ccsd[1475]: Local version # : 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote version #: 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote copy of cluster.conf is from quorate node. Jun 17 21:55:17 sklusp01a ccsd[1475]: Local version # : 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote version #: 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote copy of cluster.conf is from quorate node. Jun 17 21:55:17 sklusp01a ccsd[1475]: Local version # : 6 Jun 17 21:55:17 sklusp01a ccsd[1475]: Remote version #: 6 Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] AIS Executive Service: started and ready to provide service. Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] Using override node name sklusp01a-p Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] Using default multicast address of 239.192.108.130 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Token Timeout (30000 ms) retransmit timeout (1485 ms) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] token hold (1178 ms) retransmits before loss (20 retrans) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] join (60 ms) send_join (0 ms) consensus (60000 ms) merge (200 ms) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1402 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] send threads (0 threads) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] RRP token expired timeout (1485 ms) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] RRP token problem counter (2000 ms) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] RRP threshold (10 problem count) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] RRP mode set to none. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] heartbeat_failures_allowed (0) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] max_network_delay (50 ms) Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] The network interface [10.214.21.54] is now up. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Created or loaded sequence id 296.10.214.21.54 for this ring. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] entering GATHER state from 15. Jun 17 21:55:17 sklusp01a openais[1484]: [CMAN ] CMAN 2.0.115 (built Mar 16 2010 10:29:01) started Jun 17 21:55:17 sklusp01a openais[1484]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais extended virtual synchrony service' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais cluster membership service B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais availability management framework B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais checkpoint service B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais event service B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais distributed locking service B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais message service B.01.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais configuration service' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SERV ] Service initialized 'openais cluster config database access v1.01' Jun 17 21:55:17 sklusp01a openais[1484]: [SYNC ] Not using a virtual synchrony filter. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Creating commit token because I am the rep. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Saving state aru 0 high seq received 0 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Storing new sequence id for ring 12c Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] entering COMMIT state. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] entering RECOVERY state. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] position [0] member 10.214.21.54: Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] previous ring seq 296 rep 10.214.21.54 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] aru 0 high delivered 0 received flag 1 Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Did not need to originate any messages in recovery. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] Sending initial ORF token Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] CLM CONFIGURATION CHANGE Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] New Configuration: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] Members Left: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] Members Joined: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] CLM CONFIGURATION CHANGE Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] New Configuration: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.54) Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] Members Left: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] Members Joined: Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.54) Jun 17 21:55:17 sklusp01a openais[1484]: [SYNC ] This node is within the primary component and will provide service. Jun 17 21:55:17 sklusp01a openais[1484]: [TOTEM] entering OPERATIONAL state. Jun 17 21:55:17 sklusp01a openais[1484]: [CMAN ] quorum regained, resuming activity Jun 17 21:55:17 sklusp01a openais[1484]: [CLM ] got nodejoin message 10.214.21.54 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] entering GATHER state from 11. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] Creating commit token because I am the rep. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] Saving state aru c high seq received c Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] Storing new sequence id for ring 130 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] entering COMMIT state. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] entering RECOVERY state. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] position [0] member 10.214.21.54: Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] previous ring seq 300 rep 10.214.21.54 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] aru c high delivered c received flag 1 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] position [1] member 10.214.21.182: Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] previous ring seq 300 rep 10.214.21.182 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] aru 31 high delivered 31 received flag 1 Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] Did not need to originate any messages in recovery. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] Sending initial ORF token Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] CLM CONFIGURATION CHANGE Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] New Configuration: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.54) Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] Members Left: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] Members Joined: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] CLM CONFIGURATION CHANGE Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] New Configuration: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.54) Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.182) Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] Members Left: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] Members Joined: Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] r(0) ip(10.214.21.182) Jun 17 21:55:18 sklusp01a openais[1484]: [SYNC ] This node is within the primary component and will provide service. Jun 17 21:55:18 sklusp01a openais[1484]: [TOTEM] entering OPERATIONAL state. Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] got nodejoin message 10.214.21.54 Jun 17 21:55:18 sklusp01a openais[1484]: [CLM ] got nodejoin message 10.214.21.182 Jun 17 21:55:18 sklusp01a openais[1484]: [CPG ] got joinlist message from node 2 Jun 17 21:55:18 sklusp01a ccsd[1475]: Initial status:: Quorate Jun 17 21:55:25 sklusp01a kernel: device-mapper: dm-log-clustered: (built Aug 9 2011 11:51:11) installed Jun 17 21:55:25 sklusp01a clogd[1547]: Starting clogd: Jun 17 21:55:25 sklusp01a clogd[1547]: Built: Mar 3 2010 10:31:57 Jun 17 21:55:32 sklusp01a kernel: dlm: Using TCP for communications Jun 17 21:55:32 sklusp01a kernel: dlm: connecting to 2 Jun 17 21:55:33 sklusp01a clvmd: Cluster LVM daemon started - connected to CMAN Jun 17 21:55:44 sklusp01a clurgmgrd[1649]: Resource Group Manager Starting Jun 17 21:55:44 sklusp01a clurgmgrd: [1649]: script:l1mdt: stop of /etc/init.d/l1mdt failed (returned 1) Jun 17 21:55:44 sklusp01a clurgmgrd[1649]: stop on script "l1mdt" returned 1 (generic error) Jun 17 22:07:19 sklusp01a multipathd: dm-12: add map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-12: remove map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-12: add map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-13: add map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-14: add map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-12: add map (uevent) Jun 17 22:07:20 sklusp01a multipathd: dm-11: add map (uevent) Jun 17 22:07:20 sklusp01a dmeventd[3896]: dmeventd ready for processing. Jun 17 22:07:21 sklusp01a lvm[3896]: Monitoring mirror device vgl1mdt-lvol1 for events Jun 17 22:51:09 sklusp01a lvm[3896]: vgl1mdt-lvol1 is now in-sync Jun 17 22:52:51 sklusp01a clurgmgrd[1649]: Starting stopped service service:l1mdt Jun 17 22:52:52 sklusp01a kernel: Lustre: OBD class driver, http://www.lustre.org/ Jun 17 22:52:52 sklusp01a kernel: Lustre: Lustre Version: 1.8.5 Jun 17 22:52:52 sklusp01a kernel: Lustre: Build Version: 1.8.5-20101116203234-PRISTINE-2.6.18-194.17.1.el5_lustre.1.8.5 Jun 17 22:52:52 sklusp01a kernel: Lustre: Added LNI 10.214.127.54@tcp [8/256/0/180] Jun 17 22:52:52 sklusp01a kernel: Lustre: Accept secure, port 988 Jun 17 22:52:52 sklusp01a kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Jun 17 22:52:52 sklusp01a kernel: init dynlocks cache Jun 17 22:52:52 sklusp01a kernel: ldiskfs created from ext3-2.6-rhel5 Jun 17 22:52:52 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 17 22:52:52 sklusp01a kernel: LDISKFS-fs warning: checktime reached, running e2fsck is recommended Jun 17 22:52:52 sklusp01a kernel: LDISKFS FS on dm-11, internal journal Jun 17 22:52:52 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 17 22:52:52 sklusp01a multipathd: dm-11: umount map (uevent) Jun 17 22:52:52 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 17 22:52:52 sklusp01a kernel: LDISKFS-fs warning: checktime reached, running e2fsck is recommended Jun 17 22:52:52 sklusp01a kernel: LDISKFS FS on dm-11, internal journal Jun 17 22:52:52 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 17 22:52:52 sklusp01a kernel: Lustre: MGS MGS started Jun 17 22:52:52 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 17 22:52:52 sklusp01a kernel: Lustre: Enabling user_xattr Jun 17 22:52:52 sklusp01a kernel: Lustre: Enabling ACL Jun 17 22:52:52 sklusp01a kernel: Lustre: 11216:0:(mds_fs.c:677:mds_init_server_data()) RECOVERY: service l1-MDT0000, 56 recoverable clients, 0 delayed clients, last_transno 133173826553 Jun 17 22:52:52 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 17 22:52:52 sklusp01a kernel: Lustre: l1-MDT0000: Will be in recovery for at least 5:00, or until 56 clients reconnect Jun 17 22:52:52 sklusp01a kernel: Lustre: 11216:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 17 22:52:52 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 17 22:52:52 sklusp01a kernel: Lustre: 11216:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 17 22:52:52 sklusp01a kernel: Lustre: 11216:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0001_UUID Jun 17 22:52:52 sklusp01a kernel: Lustre: 11000:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0001_UUID Jun 17 22:52:53 sklusp01a kernel: Lustre: 11000:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0003_UUID Jun 17 22:52:53 sklusp01a kernel: LustreError: 11221:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x4e1f1f3:0xf64cbe48: rc -2 Jun 17 22:52:53 sklusp01a kernel: LustreError: 11221:0:(llog_cat.c:172:llog_cat_id2handle()) error opening log id 0x4e1f1f3:f64cbe48: rc -2 Jun 17 22:52:53 sklusp01a kernel: LustreError: 11221:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x4e1f1f3 Jun 17 22:52:53 sklusp01a clurgmgrd[1649]: Service service:l1mdt started Jun 17 22:52:57 sklusp01a kernel: Lustre: 11000:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405056578486279 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 5s ago has timed out (5s prior to deadline). Jun 17 22:52:57 sklusp01a kernel: req@ffff8111def0ec00 x1405056578486279/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339966377 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 22:52:57 sklusp01a kernel: Lustre: 11000:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405056578486281 sent from l1-OST0002-osc to NID 10.214.127.56@tcp 5s ago has timed out (5s prior to deadline). Jun 17 22:52:57 sklusp01a kernel: req@ffff8111ef4cc400 x1405056578486281/t0 o8->l1-OST0002_UUID@10.214.127.56@tcp:28/4 lens 368/584 e 0 to 1 dl 1339966377 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 22:52:58 sklusp01a kernel: Lustre: 11000:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405056578486283 sent from l1-OST0004-osc to NID 10.214.127.57@tcp 5s ago has timed out (5s prior to deadline). Jun 17 22:52:58 sklusp01a kernel: req@ffff8111d3cb5400 x1405056578486283/t0 o8->l1-OST0004_UUID@10.214.127.57@tcp:28/4 lens 368/584 e 0 to 1 dl 1339966378 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 22:52:58 sklusp01a kernel: Lustre: 11000:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0002_UUID Jun 17 22:52:58 sklusp01a kernel: Lustre: 11000:0:(mds_lov.c:1191:mds_notify()) Skipped 1 previous similar message Jun 17 22:53:07 sklusp01a kernel: Lustre: 11164:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 55 recoverable clients remain Jun 17 22:53:07 sklusp01a kernel: Lustre: 11155:0:(mds_open.c:895:mds_open_by_fid()) Orphan 1a6fb00:f64cdf9a found and opened in PENDING directory Jun 17 22:53:07 sklusp01a kernel: Lustre: 11171:0:(mds_open.c:895:mds_open_by_fid()) Orphan 1a6f455:f64baabd found and opened in PENDING directory Jun 17 22:53:08 sklusp01a kernel: Lustre: 11171:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 54 recoverable clients remain Jun 17 22:53:08 sklusp01a kernel: Lustre: 11180:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 52 recoverable clients remain Jun 17 22:53:08 sklusp01a kernel: Lustre: 11180:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 1 previous similar message Jun 17 22:53:10 sklusp01a kernel: Lustre: 11161:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 46 recoverable clients remain Jun 17 22:53:10 sklusp01a kernel: Lustre: 11161:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 5 previous similar messages Jun 17 22:53:12 sklusp01a kernel: Lustre: 11172:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 26 recoverable clients remain Jun 17 22:53:12 sklusp01a kernel: Lustre: 11172:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 19 previous similar messages Jun 17 22:53:21 sklusp01a kernel: Lustre: 11168:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 17 recoverable clients remain Jun 17 22:53:21 sklusp01a kernel: Lustre: 11168:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 8 previous similar messages Jun 17 22:53:33 sklusp01a kernel: Lustre: 11180:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 2 recoverable clients remain Jun 17 22:53:33 sklusp01a kernel: Lustre: 11180:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 14 previous similar messages Jun 17 23:03:07 sklusp01a kernel: LustreError: 11180:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.88@tcp Jun 17 23:03:07 sklusp01a kernel: LustreError: 11180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111da988000 x1405018449265568/t0 o101->@:0/0 lens 512/0 e 0 to 0 dl 1339967029 ref 1 fl Interpret:/4/0 rc -107/0 Jun 17 23:03:07 sklusp01a kernel: LustreError: 11157:0:(ldlm_lib.c:944:target_handle_connect()) l1-MDT0000: denying connection for new client 10.214.127.88@tcp (c67b1cbf-4ed5-dafb-323a-0164138a6efb): 1 clients in recovery for 0s Jun 17 23:03:07 sklusp01a kernel: LustreError: 11175:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:07 sklusp01a kernel: LustreError: 11175:0:(handler.c:1512:mds_handle()) Skipped 1 previous similar message Jun 17 23:03:07 sklusp01a kernel: LustreError: 11180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111f8b18000 x1402389212741079/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967029 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:07 sklusp01a kernel: LustreError: 11180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 17 23:03:08 sklusp01a kernel: LustreError: 11155:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:08 sklusp01a kernel: LustreError: 11175:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111bf5cc800 x1402389212742543/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967030 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11163:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111bfd6cc00 x1402389212742542/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967030 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11175:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1462 previous similar messages Jun 17 23:03:08 sklusp01a kernel: LustreError: 11163:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1462 previous similar messages Jun 17 23:03:08 sklusp01a kernel: LustreError: 11155:0:(handler.c:1512:mds_handle()) Skipped 1733 previous similar messages Jun 17 23:03:08 sklusp01a kernel: LDISKFS-fs error (device dm-11): ldiskfs_lookup: unlinked inode 27720411 in dir #29287441 Jun 17 23:03:08 sklusp01a kernel: Remounting filesystem read-only Jun 17 23:03:08 sklusp01a kernel: Lustre: 11174:0:(mds_unlink_open.c:324:mds_cleanup_pending()) l1-MDT0000: removed 2 pending open-unlinked files Jun 17 23:03:08 sklusp01a kernel: Lustre: l1-MDT0000: Post recovery failed, rc -2 Jun 17 23:03:08 sklusp01a kernel: Lustre: l1-MDT0000: Recovery period over after 10:01, of 56 clients 54 recovered and 2 were evicted. Jun 17 23:03:08 sklusp01a kernel: Lustre: l1-MDT0000: sending delayed replies to recovered clients Jun 17 23:03:08 sklusp01a kernel: LustreError: 11171:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11155:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11155:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:03:08 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1 previous similar message Jun 17 23:03:08 sklusp01a kernel: LustreError: 11171:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 67 previous similar messages Jun 17 23:03:08 sklusp01a kernel: LDISKFS-fs warning (device dm-11): kmmpd: kmmpd being stopped since filesystem has been remounted as readonly. Jun 17 23:03:09 sklusp01a kernel: LustreError: 11149:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:09 sklusp01a kernel: LustreError: 11149:0:(handler.c:1512:mds_handle()) Skipped 2611 previous similar messages Jun 17 23:03:10 sklusp01a kernel: LustreError: 11151:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111c98d1400 x1402389212748277/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967032 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:10 sklusp01a kernel: LustreError: 11151:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5814 previous similar messages Jun 17 23:03:11 sklusp01a kernel: LustreError: 11158:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:11 sklusp01a kernel: LustreError: 11158:0:(handler.c:1512:mds_handle()) Skipped 5620 previous similar messages Jun 17 23:03:13 sklusp01a kernel: LustreError: 11134:0:(llog_lvfs.c:577:llog_filp_open()) logfile creation CONFIGS/l1-client: -30 Jun 17 23:03:13 sklusp01a kernel: LustreError: 11134:0:(mgs_handler.c:672:mgs_handle()) MGS handle cmd=501 rc=-30 Jun 17 23:03:14 sklusp01a kernel: LustreError: 11178:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111cc713800 x1402389212759445/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967036 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:14 sklusp01a kernel: LustreError: 11178:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 11117 previous similar messages Jun 17 23:03:14 sklusp01a kernel: LustreError: 11156:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:03:14 sklusp01a kernel: LustreError: 11156:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 972 previous similar messages Jun 17 23:03:14 sklusp01a kernel: LustreError: 11156:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:03:15 sklusp01a kernel: LustreError: 13214:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:15 sklusp01a kernel: LustreError: 13214:0:(handler.c:1512:mds_handle()) Skipped 11051 previous similar messages Jun 17 23:03:22 sklusp01a kernel: LustreError: 13217:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111f9c7d800 x1402389212780542/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967044 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:22 sklusp01a kernel: LustreError: 13217:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 21051 previous similar messages Jun 17 23:03:23 sklusp01a kernel: LustreError: 11168:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:23 sklusp01a kernel: LustreError: 11168:0:(handler.c:1512:mds_handle()) Skipped 20870 previous similar messages Jun 17 23:03:28 sklusp01a kernel: LustreError: 11178:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:03:28 sklusp01a kernel: LustreError: 11178:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:03:38 sklusp01a kernel: LustreError: 11158:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111f9c0b400 x1402389212818815/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967060 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:03:38 sklusp01a kernel: LustreError: 11158:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38165 previous similar messages Jun 17 23:03:39 sklusp01a kernel: LustreError: 11157:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:03:39 sklusp01a kernel: LustreError: 11157:0:(handler.c:1512:mds_handle()) Skipped 37976 previous similar messages Jun 17 23:03:44 sklusp01a kernel: LustreError: 11164:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:03:44 sklusp01a kernel: LustreError: 11164:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:04:01 sklusp01a kernel: LustreError: 11170:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:04:01 sklusp01a kernel: LustreError: 11156:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:04:01 sklusp01a kernel: LustreError: 11156:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1695 previous similar messages Jun 17 23:04:01 sklusp01a kernel: LustreError: 11170:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 4 previous similar messages Jun 17 23:04:02 sklusp01a kernel: LustreError: 11158:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:04:10 sklusp01a kernel: LustreError: 11176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81121f140000 x1402389212884525/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967092 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:04:10 sklusp01a kernel: LustreError: 11176:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 65646 previous similar messages Jun 17 23:04:11 sklusp01a kernel: LustreError: 11171:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:04:11 sklusp01a kernel: LustreError: 11171:0:(handler.c:1512:mds_handle()) Skipped 65278 previous similar messages Jun 17 23:04:22 sklusp01a kernel: LustreError: 11167:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:04:22 sklusp01a kernel: LustreError: 11167:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 146 previous similar messages Jun 17 23:04:22 sklusp01a kernel: LustreError: 11167:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:04:44 sklusp01a kernel: LustreError: 11155:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:04:44 sklusp01a kernel: LustreError: 11155:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:05:01 sklusp01a kernel: LustreError: 11169:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:05:01 sklusp01a kernel: LustreError: 11169:0:(mds_reint.c:251:mds_finish_transno()) Skipped 149 previous similar messages Jun 17 23:05:08 sklusp01a kernel: LustreError: 11170:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:05:14 sklusp01a kernel: LustreError: 11178:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111aef39050 x1402389212991953/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967156 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:05:14 sklusp01a kernel: LustreError: 11178:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 107379 previous similar messages Jun 17 23:05:15 sklusp01a kernel: LustreError: 13215:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:05:15 sklusp01a kernel: LustreError: 13215:0:(handler.c:1512:mds_handle()) Skipped 107071 previous similar messages Jun 17 23:05:34 sklusp01a kernel: LustreError: 11163:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:05:34 sklusp01a kernel: LustreError: 11163:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 151 previous similar messages Jun 17 23:05:34 sklusp01a kernel: LustreError: 11163:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:06:01 sklusp01a kernel: LustreError: 11157:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:06:01 sklusp01a kernel: LustreError: 11157:0:(mds_reint.c:251:mds_finish_transno()) Skipped 148 previous similar messages Jun 17 23:06:32 sklusp01a kernel: LustreError: 13216:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:06:32 sklusp01a kernel: LustreError: 13216:0:(mds_fs.c:236:mds_client_add()) Skipped 1 previous similar message Jun 17 23:07:01 sklusp01a kernel: LustreError: 11152:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:07:01 sklusp01a kernel: LustreError: 11152:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 152 previous similar messages Jun 17 23:07:01 sklusp01a kernel: LustreError: 11152:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:07:01 sklusp01a kernel: LustreError: 11152:0:(mds_reint.c:251:mds_finish_transno()) Skipped 151 previous similar messages Jun 17 23:07:22 sklusp01a kernel: LustreError: 11171:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8111bf5cbc00 x1402389213170352/t0 o101->@:0/0 lens 296/0 e 0 to 0 dl 1339967284 ref 1 fl Interpret:/0/0 rc -107/0 Jun 17 23:07:22 sklusp01a kernel: LustreError: 11171:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 178388 previous similar messages Jun 17 23:07:23 sklusp01a kernel: LustreError: 11150:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-10.214.127.216@tcp Jun 17 23:07:23 sklusp01a kernel: LustreError: 11150:0:(handler.c:1512:mds_handle()) Skipped 178330 previous similar messages Jun 17 23:07:38 sklusp01a kernel: LustreError: 11155:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:07:38 sklusp01a kernel: LustreError: 11155:0:(mds_fs.c:236:mds_client_add()) Skipped 1 previous similar message Jun 17 23:08:01 sklusp01a kernel: LustreError: 11180:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:08:01 sklusp01a kernel: LustreError: 11180:0:(mds_reint.c:251:mds_finish_transno()) Skipped 152 previous similar messages Jun 17 23:08:55 sklusp01a kernel: LustreError: 11431:0:(llog_lvfs.c:577:llog_filp_open()) logfile creation CONFIGS/l1-client: -30 Jun 17 23:08:55 sklusp01a kernel: LustreError: 11431:0:(mgs_handler.c:672:mgs_handle()) MGS handle cmd=501 rc=-30 Jun 17 23:09:01 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:09:01 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) Skipped 141 previous similar messages Jun 17 23:09:32 sklusp01a kernel: LustreError: 11178:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:09:32 sklusp01a kernel: LustreError: 11178:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 454 previous similar messages Jun 17 23:10:01 sklusp01a kernel: LustreError: 11178:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:10:01 sklusp01a kernel: LustreError: 11178:0:(mds_reint.c:251:mds_finish_transno()) Skipped 149 previous similar messages Jun 17 23:10:14 sklusp01a kernel: LustreError: 11170:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:10:14 sklusp01a kernel: LustreError: 11170:0:(mds_fs.c:236:mds_client_add()) Skipped 4 previous similar messages Jun 17 23:11:44 sklusp01a kernel: LustreError: 11152:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111d3eaf400 x1405018450166173/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339967604 ref 1 fl Interpret:/0/0 rc -30/0 Jun 17 23:11:44 sklusp01a kernel: LustreError: 11152:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 59804 previous similar messages Jun 17 23:12:01 sklusp01a kernel: LustreError: 11167:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:12:01 sklusp01a kernel: LustreError: 11167:0:(mds_reint.c:251:mds_finish_transno()) Skipped 299 previous similar messages Jun 17 23:14:01 sklusp01a kernel: LustreError: 11155:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:14:01 sklusp01a kernel: LustreError: 11155:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 609 previous similar messages Jun 17 23:15:01 sklusp01a kernel: LustreError: 11158:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:15:01 sklusp01a kernel: LustreError: 11158:0:(mds_reint.c:251:mds_finish_transno()) Skipped 454 previous similar messages Jun 17 23:15:08 sklusp01a kernel: LustreError: 11164:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:15:08 sklusp01a kernel: LustreError: 11164:0:(mds_fs.c:236:mds_client_add()) Skipped 5 previous similar messages Jun 17 23:20:01 sklusp01a kernel: LustreError: 11160:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:20:01 sklusp01a kernel: LustreError: 11160:0:(mds_reint.c:251:mds_finish_transno()) Skipped 743 previous similar messages Jun 17 23:21:08 sklusp01a kernel: LustreError: 11153:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111c044a800 x1405018450166347/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339968168 ref 1 fl Interpret:/0/0 rc -30/0 Jun 17 23:21:08 sklusp01a kernel: LustreError: 11153:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10 previous similar messages Jun 17 23:23:01 sklusp01a kernel: LustreError: 11156:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:23:01 sklusp01a kernel: LustreError: 11156:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1349 previous similar messages Jun 17 23:24:14 sklusp01a kernel: LustreError: 11172:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:24:14 sklusp01a kernel: LustreError: 11172:0:(mds_fs.c:236:mds_client_add()) Skipped 9 previous similar messages Jun 17 23:29:01 sklusp01a kernel: LustreError: 11174:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:29:01 sklusp01a kernel: LustreError: 11174:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1349 previous similar messages Jun 17 23:31:34 sklusp01a kernel: LustreError: 11157:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111bf68b800 x1405018450166552/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339968794 ref 1 fl Interpret:/0/0 rc -30/0 Jun 17 23:31:34 sklusp01a kernel: LustreError: 11157:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27 previous similar messages Jun 17 23:33:01 sklusp01a kernel: LustreError: 11160:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:33:01 sklusp01a kernel: LustreError: 11160:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1524 previous similar messages Jun 17 23:34:24 sklusp01a kernel: LustreError: 11152:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:34:24 sklusp01a kernel: LustreError: 11152:0:(mds_fs.c:236:mds_client_add()) Skipped 33 previous similar messages Jun 17 23:39:01 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:39:01 sklusp01a kernel: LustreError: 11170:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1499 previous similar messages Jun 17 23:41:36 sklusp01a kernel: LustreError: 11153:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111f9c0e800 x1405018450166755/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339969396 ref 1 fl Interpret:/0/0 rc -30/0 Jun 17 23:41:36 sklusp01a kernel: LustreError: 11153:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25 previous similar messages Jun 17 23:43:01 sklusp01a kernel: LustreError: 11161:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:43:01 sklusp01a kernel: LustreError: 11161:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1523 previous similar messages Jun 17 23:44:36 sklusp01a kernel: LustreError: 11176:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:44:36 sklusp01a kernel: LustreError: 11176:0:(mds_fs.c:236:mds_client_add()) Skipped 22 previous similar messages Jun 17 23:49:01 sklusp01a kernel: LustreError: 13215:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 17 23:49:01 sklusp01a kernel: LustreError: 13215:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1501 previous similar messages Jun 17 23:51:41 sklusp01a kernel: LustreError: 11172:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111d1e0a800 x1402389213233141/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339970001 ref 1 fl Interpret:/0/0 rc -30/0 Jun 17 23:51:41 sklusp01a kernel: LustreError: 11172:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20 previous similar messages Jun 17 23:53:01 sklusp01a kernel: LustreError: 11180:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 17 23:53:01 sklusp01a kernel: LustreError: 11180:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1523 previous similar messages Jun 17 23:54:43 sklusp01a kernel: LustreError: 11165:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 17 23:54:43 sklusp01a kernel: LustreError: 11165:0:(mds_fs.c:236:mds_client_add()) Skipped 27 previous similar messages Jun 17 23:56:27 sklusp01a clurgmgrd[1649]: Stopping service service:l1mdt Jun 17 23:56:27 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 17 23:56:28 sklusp01a kernel: LustreError: 23539:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 17 23:56:28 sklusp01a kernel: LustreError: 23539:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 17 23:56:28 sklusp01a kernel: LustreError: 11165:0:(handler.c:1512:mds_handle()) operation 41 on unconnected MDS from 12345-10.214.127.201@tcp Jun 17 23:56:28 sklusp01a kernel: LustreError: 11165:0:(handler.c:1512:mds_handle()) Skipped 58300 previous similar messages Jun 17 23:56:28 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:56:28 sklusp01a kernel: Lustre: MGS has stopped. Jun 17 23:56:28 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:56:31 sklusp01a last message repeated 2 times Jun 17 23:56:31 sklusp01a kernel: LustreError: Skipped 3 previous similar messages Jun 17 23:56:32 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111da9da800 x1405056578487379/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:56:32 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111d38f3000 x1405056578487380/t0 o400->l1-OST0001_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:56:34 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:56:34 sklusp01a kernel: LustreError: Skipped 4 previous similar messages Jun 17 23:56:38 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:56:38 sklusp01a kernel: LustreError: Skipped 12 previous similar messages Jun 17 23:56:47 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:56:47 sklusp01a kernel: LustreError: Skipped 39 previous similar messages Jun 17 23:56:57 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111d4290000 x1405056578487385/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:56:57 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Jun 17 23:56:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 30 secs. Jun 17 23:57:03 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:57:03 sklusp01a kernel: LustreError: Skipped 53 previous similar messages Jun 17 23:57:22 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111f8b61c00 x1405056578487391/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:57:22 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Jun 17 23:57:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 60 secs. Jun 17 23:57:35 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:57:35 sklusp01a kernel: LustreError: Skipped 100 previous similar messages Jun 17 23:57:47 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111bfe30800 x1405056578487397/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:57:47 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Jun 17 23:57:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 90 secs. Jun 17 23:58:12 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111bfcce800 x1405056578487403/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:58:12 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Jun 17 23:58:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 120 secs. Jun 17 23:58:37 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111cce82800 x1405056578487409/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:58:37 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Jun 17 23:58:47 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 17 23:58:47 sklusp01a kernel: LustreError: Skipped 165 previous similar messages Jun 17 23:58:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 150 secs. Jun 17 23:59:02 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111d3c81c00 x1405056578487415/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:59:02 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Jun 17 23:59:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 180 secs. Jun 17 23:59:52 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111f9c81800 x1405056578487427/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 17 23:59:52 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 11 previous similar messages Jun 17 23:59:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 210 secs. Jun 18 00:00:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 240 secs. Jun 18 00:00:56 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 18 00:00:56 sklusp01a kernel: LustreError: Skipped 228 previous similar messages Jun 18 00:01:07 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111c044b800 x1405056578487445/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 00:01:07 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 17 previous similar messages Jun 18 00:01:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 300 secs. Jun 18 00:01:28 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 00:01:41 sklusp01a kernel: LustreError: 13215:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8111bfc51c00 x1405018406184934/t0 o38->@:0/0 lens 368/0 e 0 to 0 dl 1339970601 ref 1 fl Interpret:/0/0 rc -19/0 Jun 18 00:01:41 sklusp01a kernel: LustreError: 13215:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 763 previous similar messages Jun 18 00:02:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 390 secs. Jun 18 00:02:58 sklusp01a kernel: Lustre: Skipped 2 previous similar messages Jun 18 00:03:37 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111c044a000 x1405056578487481/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 00:03:37 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 35 previous similar messages Jun 18 00:05:12 sklusp01a kernel: LustreError: 137-5: UUID 'l1-MDT0000_UUID' is not available for connect (stopping) Jun 18 00:05:12 sklusp01a kernel: LustreError: Skipped 368 previous similar messages Jun 18 00:05:28 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 540 secs. Jun 18 00:05:28 sklusp01a kernel: Lustre: Skipped 4 previous similar messages Jun 18 00:08:12 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8111eed5c000 x1405056578487547/t0 o400->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 192/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 00:08:12 sklusp01a kernel: LustreError: 10999:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 65 previous similar messages Jun 18 00:09:58 sklusp01a kernel: Lustre: Mount still busy with 24 refs after 810 secs. Jun 18 00:09:58 sklusp01a kernel: Lustre: Skipped 8 previous similar messages Jun 18 02:16:10 sklusp01a syslogd 1.4.1: restart. Jun 18 02:16:10 sklusp01a kernel: klogd 1.4.1, log source = /proc/kmsg started. Jun 18 02:16:10 sklusp01a kernel: Linux version 2.6.18-194.17.1.el5_lustre.1.8.5 (lbuild@x86-build-1) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Nov 16 17:59:07 MST 2010 Jun 18 02:16:10 sklusp01a kernel: Command line: ro root=/dev/vg00/root acpi=off console=ttyS0,115200 console=tty0 rhgb quiet crashkernel=128M@16M Jun 18 02:16:10 sklusp01a kernel: BIOS-provided physical RAM map: Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 0000000000010000 - 000000000009f400 (usable) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 0000000000100000 - 00000000df62f000 (usable) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000df62f000 - 00000000df63c000 (ACPI data) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000df63c000 - 00000000df63d000 (usable) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000df63d000 - 00000000e4000000 (reserved) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved) Jun 18 02:16:10 sklusp01a kernel: BIOS-e820: 0000000100000000 - 000000121ffff000 (usable) Jun 18 02:16:10 sklusp01a kernel: DMI 2.7 present. Jun 18 02:16:10 sklusp01a kernel: No NUMA configuration found Jun 18 02:16:10 sklusp01a kernel: Faking a node at 0000000000000000-000000121ffff000 Jun 18 02:16:10 sklusp01a kernel: Bootmem setup node 0 0000000000000000-000000121ffff000 Jun 18 02:16:10 sklusp01a kernel: Intel MultiProcessor Specification v1.4 Jun 18 02:16:10 sklusp01a kernel: Virtual Wire compatibility mode. Jun 18 02:16:10 sklusp01a kernel: OEM ID: HP Product ID: PROLIANT APIC at: 0xFEE00000 Jun 18 02:16:10 sklusp01a kernel: Processor #16 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #0 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #2 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #4 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #6 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #18 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #20 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: Processor #22 6:10 APIC version 20 Jun 18 02:16:10 sklusp01a kernel: I/O APIC #8 Version 32 at 0xFEC00000. Jun 18 02:16:10 sklusp01a kernel: I/O APIC #0 Version 32 at 0xFEC80000. Jun 18 02:16:10 sklusp01a kernel: Setting APIC routing to clustered Jun 18 02:16:10 sklusp01a kernel: Processors: 8 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 000000000009f000 - 00000000000a0000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000000a0000 - 00000000000f0000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000000f0000 - 0000000000100000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000df62f000 - 00000000df63c000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000df63d000 - 00000000e4000000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000e4000000 - 00000000fec00000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000fec00000 - 00000000fee10000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000fee10000 - 00000000ff800000 Jun 18 02:16:10 sklusp01a kernel: Nosave address range: 00000000ff800000 - 0000000100000000 Jun 18 02:16:10 sklusp01a kernel: Allocating PCI resources starting at e6000000 (gap: e4000000:1ac00000) Jun 18 02:16:10 sklusp01a kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs Jun 18 02:16:10 sklusp01a kernel: Built 1 zonelists. Total pages: 18610533 Jun 18 02:16:10 sklusp01a kernel: Kernel command line: ro root=/dev/vg00/root acpi=off console=ttyS0,115200 console=tty0 rhgb quiet crashkernel=128M@16M Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#0 Jun 18 02:16:10 sklusp01a kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Jun 18 02:16:10 sklusp01a kernel: Console: colour VGA+ 80x25 Jun 18 02:16:10 sklusp01a kernel: Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) Jun 18 02:16:10 sklusp01a kernel: Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) Jun 18 02:16:10 sklusp01a kernel: Checking aperture... Jun 18 02:16:10 sklusp01a kernel: ACPI: DMAR not present Jun 18 02:16:10 sklusp01a kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Jun 18 02:16:10 sklusp01a kernel: Placing software IO TLB between 0x1505f000 - 0x1905f000 Jun 18 02:16:10 sklusp01a kernel: Memory: 74043944k/76021756k available (2576k kernel code, 1443024k reserved, 1306k data, 212k init) Jun 18 02:16:10 sklusp01a kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 5867.01 BogoMIPS (lpj=2933505) Jun 18 02:16:10 sklusp01a kernel: Security Framework v1.0.0 initialized Jun 18 02:16:10 sklusp01a kernel: SELinux: Initializing. Jun 18 02:16:10 sklusp01a kernel: selinux_register_security: Registering secondary module capability Jun 18 02:16:10 sklusp01a kernel: Capability LSM initialized as secondary Jun 18 02:16:10 sklusp01a kernel: Mount-cache hash table entries: 256 Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: using mwait in idle threads. Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU0: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to UP code Jun 18 02:16:10 sklusp01a kernel: ExtINT not setup in hardware but reported by MP table Jun 18 02:16:10 sklusp01a kernel: Using IO-APIC 8 Jun 18 02:16:10 sklusp01a kernel: Using IO-APIC 0 Jun 18 02:16:10 sklusp01a kernel: GSI 16 sharing vector 0xA9 and IRQ 16 Jun 18 02:16:10 sklusp01a kernel: GSI 17 sharing vector 0xB1 and IRQ 17 Jun 18 02:16:10 sklusp01a kernel: GSI 18 sharing vector 0xB9 and IRQ 18 Jun 18 02:16:10 sklusp01a kernel: GSI 19 sharing vector 0xC1 and IRQ 19 Jun 18 02:16:10 sklusp01a kernel: GSI 20 sharing vector 0xC9 and IRQ 20 Jun 18 02:16:10 sklusp01a kernel: GSI 21 sharing vector 0xD1 and IRQ 21 Jun 18 02:16:10 sklusp01a kernel: GSI 22 sharing vector 0xD9 and IRQ 22 Jun 18 02:16:10 sklusp01a kernel: GSI 23 sharing vector 0xE1 and IRQ 23 Jun 18 02:16:10 sklusp01a kernel: GSI 24 sharing vector 0xE9 and IRQ 24 Jun 18 02:16:10 sklusp01a kernel: GSI 28 sharing vector 0x32 and IRQ 28 Jun 18 02:16:10 sklusp01a kernel: GSI 30 sharing vector 0x3A and IRQ 30 Jun 18 02:16:10 sklusp01a kernel: GSI 32 sharing vector 0x42 and IRQ 32 Jun 18 02:16:10 sklusp01a kernel: GSI 34 sharing vector 0x4A and IRQ 34 Jun 18 02:16:10 sklusp01a kernel: GSI 35 sharing vector 0x52 and IRQ 35 Jun 18 02:16:10 sklusp01a kernel: GSI 36 sharing vector 0x5A and IRQ 36 Jun 18 02:16:10 sklusp01a kernel: GSI 37 sharing vector 0x62 and IRQ 37 Jun 18 02:16:10 sklusp01a kernel: GSI 38 sharing vector 0x6A and IRQ 38 Jun 18 02:16:10 sklusp01a kernel: GSI 39 sharing vector 0x72 and IRQ 39 Jun 18 02:16:10 sklusp01a kernel: GSI 40 sharing vector 0x7A and IRQ 40 Jun 18 02:16:10 sklusp01a kernel: GSI 41 sharing vector 0x82 and IRQ 41 Jun 18 02:16:10 sklusp01a kernel: GSI 42 sharing vector 0x8A and IRQ 42 Jun 18 02:16:10 sklusp01a kernel: GSI 45 sharing vector 0x92 and IRQ 45 Jun 18 02:16:10 sklusp01a kernel: GSI 47 sharing vector 0x9A and IRQ 47 Jun 18 02:16:10 sklusp01a kernel: Using local APIC timer interrupts. Jun 18 02:16:10 sklusp01a kernel: Detected 8.333 MHz APIC timer. Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 1/8 APIC 0x0 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#1 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.15 BogoMIPS (lpj=2933078) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU1: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 2/8 APIC 0x2 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#2 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.15 BogoMIPS (lpj=2933078) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU2: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 3/8 APIC 0x4 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#3 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5865.94 BogoMIPS (lpj=2932971) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 2 Jun 18 02:16:10 sklusp01a kernel: CPU3: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 4/8 APIC 0x6 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#4 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.16 BogoMIPS (lpj=2933080) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 0 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 3 Jun 18 02:16:10 sklusp01a kernel: CPU4: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 5/8 APIC 0x12 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#5 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.24 BogoMIPS (lpj=2933123) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU5: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 6/8 APIC 0x14 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#6 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.07 BogoMIPS (lpj=2933037) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 2 Jun 18 02:16:10 sklusp01a kernel: CPU6: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: SMP alternatives: switching to SMP code Jun 18 02:16:10 sklusp01a kernel: Booting processor 7/8 APIC 0x16 Jun 18 02:16:10 sklusp01a kernel: Initializing CPU#7 Jun 18 02:16:10 sklusp01a kernel: Calibrating delay using timer specific routine.. 5866.03 BogoMIPS (lpj=2933016) Jun 18 02:16:10 sklusp01a kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Jun 18 02:16:10 sklusp01a kernel: CPU: L2 cache: 256K Jun 18 02:16:10 sklusp01a kernel: CPU: L3 cache: 8192K Jun 18 02:16:10 sklusp01a kernel: CPU: Physical Processor ID: 1 Jun 18 02:16:10 sklusp01a kernel: CPU: Processor Core ID: 3 Jun 18 02:16:10 sklusp01a kernel: CPU7: Thermal monitoring enabled (TM1) Jun 18 02:16:10 sklusp01a kernel: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping 05 Jun 18 02:16:10 sklusp01a kernel: Brought up 8 CPUs Jun 18 02:16:10 sklusp01a kernel: testing NMI watchdog ... OK. Jun 18 02:16:10 sklusp01a kernel: time.c: Using 1.193182 MHz WALL PIT GTOD PIT/TSC timer. Jun 18 02:16:10 sklusp01a kernel: time.c: Detected 2933.505 MHz processor. Jun 18 02:16:10 sklusp01a kernel: migration_cost=28,133 Jun 18 02:16:10 sklusp01a kernel: checking if image is initramfs... it is Jun 18 02:16:10 sklusp01a kernel: Freeing initrd memory: 3856k freed Jun 18 02:16:10 sklusp01a kernel: NET: Registered protocol family 16 Jun 18 02:16:10 sklusp01a kernel: PCI: Using configuration type 1 Jun 18 02:16:10 sklusp01a kernel: ACPI: Interpreter disabled. Jun 18 02:16:10 sklusp01a kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Jun 18 02:16:10 sklusp01a kernel: pnp: PnP ACPI: disabled Jun 18 02:16:10 sklusp01a kernel: usbcore: registered new driver usbfs Jun 18 02:16:10 sklusp01a kernel: usbcore: registered new driver hub Jun 18 02:16:10 sklusp01a kernel: PCI: Probing PCI hardware Jun 18 02:16:10 sklusp01a kernel: ACPI Error (tbget-0168): Invalid address flags 8 [20060707] Jun 18 02:16:10 sklusp01a last message repeated 46 times Jun 18 02:16:10 sklusp01a kernel: PCI: Transparent bridge - 0000:00:1e.0 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1c.0[A] -> IRQ 169 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1d.0[A] -> IRQ 201 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1d.1[B] -> IRQ 225 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1d.2[C] -> IRQ 217 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1d.3[D] -> IRQ 225 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:00:1d.7[A] -> IRQ 201 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.0[A] -> IRQ 50 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.1[B] -> IRQ 122 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.2[B] -> IRQ 122 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.3[C] -> IRQ 130 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.4[C] -> IRQ 130 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.5[D] -> IRQ 154 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.6[D] -> IRQ 154 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:02:00.7[A] -> IRQ 50 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:06:00.0[A] -> IRQ 58 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:06:00.1[B] -> IRQ 98 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:0c:00.0[A] -> IRQ 169 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:01:03.0[A] -> IRQ 225 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:01:04.0[A] -> IRQ 209 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:01:04.2[B] -> IRQ 217 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:01:04.4[B] -> IRQ 217 Jun 18 02:16:10 sklusp01a kernel: PCI->APIC IRQ transform: 0000:01:04.6[A] -> IRQ 209 Jun 18 02:16:10 sklusp01a kernel: NetLabel: Initializing Jun 18 02:16:10 sklusp01a kernel: NetLabel: domain hash size = 128 Jun 18 02:16:10 sklusp01a kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jun 18 02:16:10 sklusp01a kernel: NetLabel: unlabeled traffic allowed by default Jun 18 02:16:10 sklusp01a kernel: ACPI: DMAR not present Jun 18 02:16:10 sklusp01a kernel: PCI-GART: No AMD northbridge found. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:01.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: f3800000-fb7fffff Jun 18 02:16:10 sklusp01a kernel: PREFETCH window 0x00000000e6000000-0x00000000e60fffff Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:02.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: disabled. Jun 18 02:16:10 sklusp01a kernel: PREFETCH window: disabled. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:03.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: disabled. Jun 18 02:16:10 sklusp01a kernel: PREFETCH window: disabled. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:07.0 Jun 18 02:16:10 sklusp01a kernel: IO window: 4000-4fff Jun 18 02:16:10 sklusp01a kernel: MEM window: fb900000-fbcfffff Jun 18 02:16:10 sklusp01a kernel: PREFETCH window 0x00000000e6100000-0x00000000e61fffff Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:08.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: disabled. Jun 18 02:16:10 sklusp01a kernel: PREFETCH window: disabled. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:09.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: disabled. Jun 18 02:16:10 sklusp01a kernel: PREFETCH window: disabled. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:0a.0 Jun 18 02:16:10 sklusp01a kernel: IO window: disabled. Jun 18 02:16:10 sklusp01a kernel: MEM window: disabled. Jun 18 02:16:10 sklusp01a kernel: PREFETCH window: disabled. Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:1c.0 Jun 18 02:16:10 sklusp01a kernel: IO window: 5000-5fff Jun 18 02:16:10 sklusp01a kernel: MEM window: fbd00000-fbffffff Jun 18 02:16:10 sklusp01a kernel: PREFETCH window 0x00000000e6200000-0x00000000e62fffff Jun 18 02:16:10 sklusp01a kernel: PCI: Bridge: 0000:00:1e.0 Jun 18 02:16:10 sklusp01a kernel: IO window: 2000-3fff Jun 18 02:16:10 sklusp01a kernel: MEM window: f3600000-f37fffff Jun 18 02:16:10 sklusp01a kernel: PREFETCH window 0x00000000e8000000-0x00000000efffffff Jun 18 02:16:10 sklusp01a kernel: NET: Registered protocol family 2 Jun 18 02:16:10 sklusp01a kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Jun 18 02:16:10 sklusp01a kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Jun 18 02:16:10 sklusp01a kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jun 18 02:16:10 sklusp01a kernel: TCP: Hash tables configured (established 262144 bind 65536) Jun 18 02:16:10 sklusp01a kernel: TCP reno registered Jun 18 02:16:10 sklusp01a kernel: audit: initializing netlink socket (disabled) Jun 18 02:16:10 sklusp01a kernel: type=2000 audit(1339978508.393:1): initialized Jun 18 02:16:10 sklusp01a kernel: Total HugeTLB memory allocated, 0 Jun 18 02:16:10 sklusp01a kernel: VFS: Disk quotas dquot_6.5.1 Jun 18 02:16:10 sklusp01a kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jun 18 02:16:10 sklusp01a kernel: Initializing Cryptographic API Jun 18 02:16:10 sklusp01a kernel: alg: No test for crc32c (crc32c-generic) Jun 18 02:16:10 sklusp01a kernel: ksign: Installing public key data Jun 18 02:16:10 sklusp01a kernel: Loading keyring Jun 18 02:16:10 sklusp01a kernel: io scheduler noop registered Jun 18 02:16:10 sklusp01a kernel: io scheduler anticipatory registered Jun 18 02:16:10 sklusp01a kernel: io scheduler deadline registered (default) Jun 18 02:16:10 sklusp01a kernel: io scheduler cfq registered Jun 18 02:16:10 sklusp01a kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jun 18 02:16:10 sklusp01a kernel: Real Time Clock Driver v1.12ac Jun 18 02:16:10 sklusp01a kernel: Non-volatile memory driver v1.2 Jun 18 02:16:10 sklusp01a kernel: Linux agpgart interface v0.101 (c) Dave Jones Jun 18 02:16:10 sklusp01a kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Jun 18 02:16:10 sklusp01a kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jun 18 02:16:10 sklusp01a kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jun 18 02:16:10 sklusp01a kernel: brd: module loaded Jun 18 02:16:10 sklusp01a kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Jun 18 02:16:10 sklusp01a kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Jun 18 02:16:10 sklusp01a kernel: ide-floppy driver 0.99.newide Jun 18 02:16:10 sklusp01a kernel: usbcore: registered new driver hiddev Jun 18 02:16:10 sklusp01a kernel: usbcore: registered new driver usbhid Jun 18 02:16:10 sklusp01a kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Jun 18 02:16:10 sklusp01a kernel: PNP: No PS/2 controller found. Probing ports directly. Jun 18 02:16:10 sklusp01a kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jun 18 02:16:10 sklusp01a kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jun 18 02:16:10 sklusp01a kernel: mice: PS/2 mouse device common for all mice Jun 18 02:16:10 sklusp01a kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 Jun 18 02:16:10 sklusp01a kernel: md: bitmap version 4.39 Jun 18 02:16:10 sklusp01a kernel: TCP bic registered Jun 18 02:16:10 sklusp01a kernel: Initializing IPsec netlink socket Jun 18 02:16:10 sklusp01a kernel: NET: Registered protocol family 1 Jun 18 02:16:10 sklusp01a kernel: NET: Registered protocol family 17 Jun 18 02:16:10 sklusp01a kernel: Initalizing network drop monitor service Jun 18 02:16:10 sklusp01a kernel: Freeing unused kernel memory: 212k freed Jun 18 02:16:10 sklusp01a kernel: Write protecting the kernel read-only data: 505k Jun 18 02:16:10 sklusp01a kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1 Jun 18 02:16:10 sklusp01a kernel: ehci_hcd 0000:00:1d.7: debug port 1 Jun 18 02:16:10 sklusp01a kernel: ehci_hcd 0000:00:1d.7: irq 201, io mem 0xf35f0000 Jun 18 02:16:10 sklusp01a kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Jun 18 02:16:10 sklusp01a kernel: usb usb1: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 1-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 1-0:1.0: 8 ports detected Jun 18 02:16:10 sklusp01a kernel: USB Universal Host Controller Interface driver v3.0 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.0: irq 201, io base 0x00001000 Jun 18 02:16:10 sklusp01a kernel: usb usb2: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 2-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 2-0:1.0: 2 ports detected Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.1: irq 225, io base 0x00001020 Jun 18 02:16:10 sklusp01a kernel: usb usb3: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 3-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 3-0:1.0: 2 ports detected Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.2: irq 217, io base 0x00001040 Jun 18 02:16:10 sklusp01a kernel: usb usb4: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 4-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 4-0:1.0: 2 ports detected Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.3: UHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:00:1d.3: irq 225, io base 0x00001060 Jun 18 02:16:10 sklusp01a kernel: usb usb5: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 5-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 5-0:1.0: 2 ports detected Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:01:04.4: UHCI Host Controller Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:01:04.4: new USB bus registered, assigned bus number 6 Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports Jun 18 02:16:10 sklusp01a kernel: uhci_hcd 0000:01:04.4: irq 217, io base 0x00003800 Jun 18 02:16:10 sklusp01a kernel: usb usb6: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: hub 6-0:1.0: USB hub found Jun 18 02:16:10 sklusp01a kernel: hub 6-0:1.0: 2 ports detected Jun 18 02:16:10 sklusp01a kernel: SCSI subsystem initialized Jun 18 02:16:10 sklusp01a kernel: HP CISS Driver (v 3.6.20-RH4) Jun 18 02:16:10 sklusp01a kernel: cciss0: <0x323a> at PCI 0000:0c:00.0 IRQ 59 using DAC Jun 18 02:16:10 sklusp01a kernel: blocks= 286679925 block_size= 512 Jun 18 02:16:10 sklusp01a kernel: heads= 255, sectors= 63, cylinders= 17845 Jun 18 02:16:10 sklusp01a kernel: Jun 18 02:16:10 sklusp01a kernel: blocks= 286679925 block_size= 512 Jun 18 02:16:10 sklusp01a kernel: heads= 255, sectors= 63, cylinders= 17845 Jun 18 02:16:10 sklusp01a kernel: Jun 18 02:16:10 sklusp01a kernel: cciss/c0d0: p1 p2 p3 Jun 18 02:16:10 sklusp01a kernel: QLogic Fibre Channel HBA Driver Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Found an ISP2532, irq 58, iobase 0xffffc20000014000 Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Configuring PCI space... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Configure NVRAM parameters... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Verifying loaded RISC code... Jun 18 02:16:10 sklusp01a kernel: usb 6-1: new full speed USB device using uhci_hcd and address 2 Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Allocated (64 KB) for EFT... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Allocated (1414 KB) for firmware dump... Jun 18 02:16:10 sklusp01a kernel: scsi0 : qla2xxx Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: Jun 18 02:16:10 sklusp01a kernel: QLogic Fibre Channel HBA Driver: 8.03.01.04.05.05-k Jun 18 02:16:10 sklusp01a kernel: QLogic QMH2562 - PCI-Express Dual Channel 8Gb Fibre Channel Mezzanine HBA Jun 18 02:16:10 sklusp01a kernel: ISP2532: PCIe (2.5Gb/s x8) @ 0000:06:00.0 hdma+, host#=0, fw=5.03.02 (d5) Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Found an ISP2532, irq 98, iobase 0xffffc2000017c000 Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Configuring PCI space... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Configure NVRAM parameters... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Verifying loaded RISC code... Jun 18 02:16:10 sklusp01a kernel: usb 6-1: configuration #1 chosen from 1 choice Jun 18 02:16:10 sklusp01a kernel: input: HP Virtual Keyboard as /class/input/input0 Jun 18 02:16:10 sklusp01a kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1 Jun 18 02:16:10 sklusp01a kernel: input: HP Virtual Keyboard as /class/input/input1 Jun 18 02:16:10 sklusp01a kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1 Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Allocated (64 KB) for EFT... Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Allocated (1414 KB) for firmware dump... Jun 18 02:16:10 sklusp01a kernel: scsi1 : qla2xxx Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: Jun 18 02:16:10 sklusp01a kernel: QLogic Fibre Channel HBA Driver: 8.03.01.04.05.05-k Jun 18 02:16:10 sklusp01a kernel: QLogic QMH2562 - PCI-Express Dual Channel 8Gb Fibre Channel Mezzanine HBA Jun 18 02:16:10 sklusp01a kernel: ISP2532: PCIe (2.5Gb/s x8) @ 0000:06:00.1 hdma+, host#=1, fw=5.03.02 (d5) Jun 18 02:16:10 sklusp01a kernel: Initializing USB Mass Storage driver... Jun 18 02:16:10 sklusp01a kernel: usbcore: registered new driver usb-storage Jun 18 02:16:10 sklusp01a kernel: USB Mass Storage support registered. Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.0: LOOP UP detected (8 Gbps). Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sda: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sda: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sda: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sda: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sda: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sda: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sda: sda1 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:0:0: Attached scsi disk sda Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdb: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdb: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdb: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdb: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdb: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdb: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdb: sdb1 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:0:1: Attached scsi disk sdb Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdc: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sdc: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdc: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdc: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sdc: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdc: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdc: sdc1 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:1:0: Attached scsi disk sdc Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdd: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdd: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdd: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdd: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdd: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdd: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdd: unknown partition table Jun 18 02:16:10 sklusp01a kernel: sd 0:0:1:1: Attached scsi disk sdd Jun 18 02:16:10 sklusp01a kernel: qla2xxx 0000:06:00.1: LOOP UP detected (8 Gbps). Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sde: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sde: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sde: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sde: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sde: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sde: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sde: sde1 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:0:0: Attached scsi disk sde Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdf: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdf: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdf: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdf: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdf: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdf: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdf: sdf1 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:0:1: Attached scsi disk sdf Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdg: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sdg: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdg: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdg: 655441920 512-byte hdwr sectors (335586 MB) Jun 18 02:16:10 sklusp01a kernel: sdg: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdg: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdg: sdg1 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:1:0: Attached scsi disk sdg Jun 18 02:16:10 sklusp01a kernel: Vendor: HP Model: OPEN-V Rev: 6008 Jun 18 02:16:10 sklusp01a kernel: Type: Direct-Access ANSI SCSI revision: 03 Jun 18 02:16:10 sklusp01a kernel: SCSI device sdh: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdh: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdh: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: SCSI device sdh: 107520 512-byte hdwr sectors (55 MB) Jun 18 02:16:10 sklusp01a kernel: sdh: Write Protect is off Jun 18 02:16:10 sklusp01a kernel: SCSI device sdh: drive cache: write back Jun 18 02:16:10 sklusp01a kernel: sdh: unknown partition table Jun 18 02:16:10 sklusp01a kernel: sd 1:0:1:1: Attached scsi disk sdh Jun 18 02:16:10 sklusp01a kernel: device-mapper: uevent: version 1.0.3 Jun 18 02:16:10 sklusp01a kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com Jun 18 02:16:10 sklusp01a kernel: device-mapper: dm-raid45: initialized v0.2594l Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: write access will be enabled during recovery. Jun 18 02:16:10 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: dm-0: orphan cleanup on readonly fs Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: dm-0: 1 orphan inode deleted Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: recovery complete. Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: mounted filesystem with ordered data mode. Jun 18 02:16:10 sklusp01a kernel: SELinux: Disabled at runtime. Jun 18 02:16:10 sklusp01a kernel: type=1404 audit(1339978539.639:2): selinux=0 auid=4294967295 ses=4294967295 Jun 18 02:16:10 sklusp01a kernel: input: PC Speaker as /class/input/input2 Jun 18 02:16:10 sklusp01a kernel: Floppy drive(s): fd0 is 1.44M Jun 18 02:16:10 sklusp01a kernel: Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver bnx2x 1.52.1-6 (2010/02/16) Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth0: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem fb000000, IRQ 50, node addr 0017a4772860 Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth1: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem fa000000, IRQ 122, node addr 0017a4772862 Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth2: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f9000000, IRQ 122, node addr 0017a4772864 Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:1:0: Attached scsi generic sg2 type 0 Jun 18 02:16:10 sklusp01a kernel: sd 0:0:1:1: Attached scsi generic sg3 type 0 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:0:0: Attached scsi generic sg4 type 0 Jun 18 02:16:10 sklusp01a kernel: eth3: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f8000000, IRQ 130, <5>sd 1:0:0:1: Attached scsi generic sg5 type 0 Jun 18 02:16:10 sklusp01a kernel: node addr 0017a4772866 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:1:0: Attached scsi generic sg6 type 0 Jun 18 02:16:10 sklusp01a kernel: sd 1:0:1:1: Attached scsi generic sg7 type 0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth4: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f7000000, IRQ 130, node addr 0017a4772868 Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth5: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f6000000, IRQ 154, node addr 0017a477286a Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth6: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f5000000, IRQ 154, node addr 0017a477286c Jun 18 02:16:10 sklusp01a kernel: bnx2x: part number 412F4E-0-0-0 Jun 18 02:16:10 sklusp01a kernel: bnx2x: Loading bnx2x-e1h-5.2.13.0.fw Jun 18 02:16:10 sklusp01a kernel: eth7: Broadcom NetXtreme II BCM57711E XGb (A0) PCI-E x4 5GHz (Gen2) found at mem f4000000, IRQ 50, node addr 0017a477286e Jun 18 02:16:10 sklusp01a kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Jun 18 02:16:10 sklusp01a kernel: floppy0: no floppy controllers found Jun 18 02:16:10 sklusp01a kernel: Floppy drive(s): fd0 is 1.44M Jun 18 02:16:10 sklusp01a kernel: floppy0: no floppy controllers found Jun 18 02:16:10 sklusp01a kernel: lp: driver loaded but no devices found Jun 18 02:16:10 sklusp01a kernel: md: Autodetecting RAID arrays. Jun 18 02:16:10 sklusp01a kernel: md: autorun ... Jun 18 02:16:10 sklusp01a kernel: md: ... autorun DONE. Jun 18 02:16:10 sklusp01a kernel: device-mapper: multipath: version 1.0.5 loaded Jun 18 02:16:10 sklusp01a kernel: device-mapper: multipath round-robin: version 1.0.0 loaded Jun 18 02:16:10 sklusp01a kernel: EXT3 FS on dm-0, internal journal Jun 18 02:16:10 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 02:16:10 sklusp01a kernel: EXT3 FS on cciss/c0d0p1, internal journal Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: mounted filesystem with ordered data mode. Jun 18 02:16:10 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 02:16:10 sklusp01a kernel: EXT3-fs warning: checktime reached, running e2fsck is recommended Jun 18 02:16:10 sklusp01a kernel: EXT3 FS on dm-1, internal journal Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: recovery complete. Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: mounted filesystem with ordered data mode. Jun 18 02:16:10 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 02:16:10 sklusp01a kernel: EXT3-fs warning: checktime reached, running e2fsck is recommended Jun 18 02:16:10 sklusp01a kernel: EXT3 FS on dm-2, internal journal Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: recovery complete. Jun 18 02:16:10 sklusp01a kernel: EXT3-fs: mounted filesystem with ordered data mode. Jun 18 02:16:10 sklusp01a kernel: EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended Jun 18 02:16:10 sklusp01a kernel: Adding 16779884k swap on /dev/cciss/c0d0p2. Priority:-1 extents:1 across:16779884k Jun 18 02:16:10 sklusp01a kernel: IA-32 Microcode Update Driver: v1.14a Jun 18 02:16:10 sklusp01a kernel: IPv6: Loaded, but administratively disabled, reboot required to enable Jun 18 02:16:10 sklusp01a kernel: Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008) Jun 18 02:16:10 sklusp01a kernel: bonding: MII link monitoring set to 100 ms Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: Adding slave eth2. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth2: using MSI-X IRQs: sp 107 fp[0] 123 ... fp[2] 139 Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: enslaving eth2 as a backup interface with a down link. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth2 NIC Link is Down Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: Adding slave eth3. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth3: using MSI-X IRQs: sp 147 fp[0] 163 ... fp[2] 179 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth2 NIC Link is Down Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth2 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a last message repeated 4 times Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: enslaving eth3 as a backup interface with a down link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: link status definitely up for interface eth2. Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: making interface eth2 the new active one. Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: first active interface up! Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: link status definitely up for interface eth3. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth3 NIC Link is Down Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: link status definitely down for interface eth3, disabling it Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth3 NIC Link is Down Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth3 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond0: link status definitely up for interface eth3. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth3 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a last message repeated 3 times Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: Adding slave eth4. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth4: using MSI-X IRQs: sp 187 fp[0] 203 ... fp[2] 219 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth4 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: Warning: failed to get speed and duplex from eth4, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: making interface eth4 the new active one. Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: first active interface up! Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: enslaving eth4 as an active interface with an up link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: Adding slave eth5. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth5: using MSI-X IRQs: sp 227 fp[0] 52 ... fp[2] 68 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth5 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: Warning: failed to get speed and duplex from eth5, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond1: enslaving eth5 as a backup interface with an up link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: Adding slave eth6. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth6: using MSI-X IRQs: sp 76 fp[0] 92 ... fp[2] 108 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth6 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: Warning: failed to get speed and duplex from eth6, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: making interface eth6 the new active one. Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: first active interface up! Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: enslaving eth6 as an active interface with an up link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: Adding slave eth7. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth7: using MSI-X IRQs: sp 116 fp[0] 132 ... fp[2] 148 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth7 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: Warning: failed to get speed and duplex from eth7, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond2: enslaving eth7 as a backup interface with an up link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: Adding slave eth0. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth0: using MSI-X IRQs: sp 156 fp[0] 172 ... fp[2] 188 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth0 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: Warning: failed to get speed and duplex from eth0, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: making interface eth0 the new active one. Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: first active interface up! Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: enslaving eth0 as an active interface with an up link. Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: Adding slave eth1. Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth1: using MSI-X IRQs: sp 196 fp[0] 212 ... fp[2] 228 Jun 18 02:16:10 sklusp01a kernel: bnx2x: eth1 NIC Link is Up, 2500 Mbps full duplex, receive & transmit flow control ON Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: Warning: failed to get speed and duplex from eth1, assumed to be 100Mb/sec and Full. Jun 18 02:16:10 sklusp01a kernel: bonding: bond3: enslaving eth1 as a backup interface with an up link. Jun 18 02:16:10 sklusp01a rpc.statd[4477]: Version 1.0.9 Starting Jun 18 02:16:11 sklusp01a kdump: kexec: loaded kdump kernel Jun 18 02:16:11 sklusp01a kdump: started up Jun 18 02:16:20 sklusp01a kernel: i2c /dev entries driver Jun 18 02:16:20 sklusp01a snmpd[4709]: could not open /proc/net/if_inet6 Jun 18 02:16:21 sklusp01a snmpd[4709]: cannot open /proc/net/snmp6 ... Jun 18 00:16:24 sklusp01a ntpdate[4821]: step time server 10.3.42.6 offset -7199.983041 sec Jun 18 00:16:24 sklusp01a ntpd[4825]: ntpd 4.2.2p1@1.1570-o Thu Nov 26 11:34:34 UTC 2009 (1) Jun 18 00:16:24 sklusp01a ntpd[4826]: precision = 1.000 usec Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface wildcard, 0.0.0.0#123 Disabled Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface lo, 127.0.0.1#123 Enabled Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface bond0, 10.214.16.54#123 Enabled Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface bond1, 10.214.19.54#123 Enabled Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface bond2, 10.214.21.54#123 Enabled Jun 18 00:16:24 sklusp01a ntpd[4826]: Listening on interface bond3, 10.214.127.54#123 Enabled Jun 18 00:16:24 sklusp01a ntpd[4826]: kernel time sync status 0040 Jun 18 00:16:24 sklusp01a ntpd[4826]: frequency initialized 7.276 PPM from /var/lib/ntp/drift Jun 18 00:16:27 sklusp01a kernel: ipmi message handler version 39.1 Jun 18 00:16:27 sklusp01a kernel: ipmi message handler version 39.1 Jun 18 00:16:27 sklusp01a kernel: ipmi device interface Jun 18 00:16:27 sklusp01a kernel: IPMI System Interface driver. Jun 18 00:16:27 sklusp01a kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 Jun 18 00:16:27 sklusp01a kernel: ipmi: Found new BMC (man_id: 0x00000b, prod_id: 0x2020, dev_id: 0x11) Jun 18 00:16:27 sklusp01a kernel: IPMI kcs interface initialized Jun 18 00:16:27 sklusp01a kernel: ipmi_si: Trying PCI-specified kcs state machine at mem address 0xf36f0000, slave address 0x0, irq 209 Jun 18 00:16:27 sklusp01a kernel: Using irq 209 Jun 18 00:16:27 sklusp01a kernel: ipmi: interfacing existing BMC (man_id: 0x00000b, prod_id: 0x2020, dev_id: 0x11) Jun 18 00:16:27 sklusp01a kernel: IPMI kcs interface initialized Jun 18 00:16:28 sklusp01a hpasmxld[5226]: hpDeferSPDThread: Starting thread to collect DIMM SPD Data. Jun 18 00:16:28 sklusp01a kernel: IPMI Watchdog: driver initialized Jun 18 00:16:29 sklusp01a hpasrd[5325]: Starting with poll 1 and timeout 600. Jun 18 00:16:29 sklusp01a hpasrd[5325]: Setting the watchdog timer. Jun 18 00:16:29 sklusp01a hpasrd[5325]: Preferred watchdog driver not found. Jun 18 00:16:29 sklusp01a hpasrd[5325]: Found iLO memory at 0xf37e0000. Jun 18 00:16:29 sklusp01a hpasrd[5325]: Successfully mapped device. Jun 18 00:16:30 sklusp01a kernel: Fusion MPT base driver 3.04.13rh Jun 18 00:16:30 sklusp01a kernel: Copyright (c) 1999-2008 LSI Corporation Jun 18 00:16:30 sklusp01a kernel: Fusion MPT misc device (ioctl) driver 3.04.13rh Jun 18 00:16:30 sklusp01a kernel: mptctl: Registered with Fusion MPT base driver Jun 18 00:16:30 sklusp01a kernel: mptctl: /dev/mptctl @ (major,minor=10,220) Jun 18 00:16:32 sklusp01a modclusterd: startup succeeded Jun 18 00:16:32 sklusp01a oddjobd: oddjobd startup succeeded Jun 18 00:16:32 sklusp01a saslauthd[8089]: detach_tty : master pid is: 8089 Jun 18 00:16:32 sklusp01a saslauthd[8089]: ipc_init : listening on socket: /var/run/saslauthd/mux Jun 18 00:16:32 sklusp01a ricci: startup succeeded Jun 18 00:17:12 sklusp01a kernel: DLM (built Nov 16 2010 17:59:42) installed Jun 18 00:17:12 sklusp01a kernel: GFS2 (built Nov 16 2010 18:00:55) installed Jun 18 00:17:12 sklusp01a kernel: Lock_DLM (built Nov 16 2010 18:01:06) installed Jun 18 00:17:12 sklusp01a ccsd[8575]: Starting ccsd 2.0.115: Jun 18 00:17:12 sklusp01a ccsd[8575]: Built: Mar 16 2010 10:28:57 Jun 18 00:17:12 sklusp01a ccsd[8575]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Jun 18 00:17:13 sklusp01a ccsd[8575]: cluster.conf (cluster name = lusp01cl, version = 6) found. Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote copy of cluster.conf is from quorate node. Jun 18 00:17:13 sklusp01a ccsd[8575]: Local version # : 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote version #: 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote copy of cluster.conf is from quorate node. Jun 18 00:17:13 sklusp01a ccsd[8575]: Local version # : 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote version #: 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote copy of cluster.conf is from quorate node. Jun 18 00:17:13 sklusp01a ccsd[8575]: Local version # : 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote version #: 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote copy of cluster.conf is from quorate node. Jun 18 00:17:13 sklusp01a ccsd[8575]: Local version # : 6 Jun 18 00:17:13 sklusp01a ccsd[8575]: Remote version #: 6 Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] AIS Executive Service: started and ready to provide service. Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] Using override node name sklusp01a-p Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] Using default multicast address of 239.192.108.130 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Token Timeout (30000 ms) retransmit timeout (1485 ms) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] token hold (1178 ms) retransmits before loss (20 retrans) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] join (60 ms) send_join (0 ms) consensus (60000 ms) merge (200 ms) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1402 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] send threads (0 threads) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] RRP token expired timeout (1485 ms) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] RRP token problem counter (2000 ms) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] RRP threshold (10 problem count) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] RRP mode set to none. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] heartbeat_failures_allowed (0) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] max_network_delay (50 ms) Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] The network interface [10.214.21.54] is now up. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Created or loaded sequence id 304.10.214.21.54 for this ring. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering GATHER state from 15. Jun 18 00:17:13 sklusp01a openais[8586]: [CMAN ] CMAN 2.0.115 (built Mar 16 2010 10:29:01) started Jun 18 00:17:13 sklusp01a openais[8586]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais extended virtual synchrony service' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais cluster membership service B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais availability management framework B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais checkpoint service B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais event service B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais distributed locking service B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais message service B.01.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais configuration service' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SERV ] Service initialized 'openais cluster config database access v1.01' Jun 18 00:17:13 sklusp01a openais[8586]: [SYNC ] Not using a virtual synchrony filter. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Creating commit token because I am the rep. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Saving state aru 0 high seq received 0 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Storing new sequence id for ring 134 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering COMMIT state. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering RECOVERY state. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] position [0] member 10.214.21.54: Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] previous ring seq 304 rep 10.214.21.54 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] aru 0 high delivered 0 received flag 1 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Did not need to originate any messages in recovery. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Sending initial ORF token Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:17:13 sklusp01a openais[8586]: [SYNC ] This node is within the primary component and will provide service. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering OPERATIONAL state. Jun 18 00:17:13 sklusp01a openais[8586]: [CMAN ] quorum regained, resuming activity Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.54 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering GATHER state from 11. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Creating commit token because I am the rep. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Saving state aru c high seq received c Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Storing new sequence id for ring 138 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering COMMIT state. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering RECOVERY state. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] position [0] member 10.214.21.54: Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] previous ring seq 308 rep 10.214.21.54 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] aru c high delivered c received flag 1 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] position [1] member 10.214.21.182: Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] previous ring seq 308 rep 10.214.21.182 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] aru 22 high delivered 22 received flag 1 Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Did not need to originate any messages in recovery. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] Sending initial ORF token Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.182) Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.182) Jun 18 00:17:13 sklusp01a openais[8586]: [SYNC ] This node is within the primary component and will provide service. Jun 18 00:17:13 sklusp01a openais[8586]: [TOTEM] entering OPERATIONAL state. Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.54 Jun 18 00:17:13 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.182 Jun 18 00:17:13 sklusp01a openais[8586]: [CPG ] got joinlist message from node 2 Jun 18 00:17:13 sklusp01a ccsd[8575]: Initial status:: Quorate Jun 18 00:17:33 sklusp01a hpasmxld[5226]: hpDeferSPDThread: End of Collecting DIMM SPD data. Jun 18 00:19:19 sklusp01a kernel: device-mapper: dm-log-clustered: (built Aug 9 2011 11:51:11) installed Jun 18 00:19:19 sklusp01a clogd[8975]: Starting clogd: Jun 18 00:19:19 sklusp01a clogd[8975]: Built: Mar 3 2010 10:31:57 Jun 18 00:19:26 sklusp01a kernel: dlm: Using TCP for communications Jun 18 00:19:26 sklusp01a kernel: dlm: connecting to 2 Jun 18 00:19:27 sklusp01a clvmd: Cluster LVM daemon started - connected to CMAN Jun 18 00:19:41 sklusp01a ntpd[4826]: synchronized to 10.3.42.6, stratum 2 Jun 18 00:19:41 sklusp01a ntpd[4826]: kernel time sync enabled 0001 Jun 18 00:21:34 sklusp01a clurgmgrd[9379]: Resource Group Manager Starting Jun 18 00:21:35 sklusp01a clurgmgrd: [9379]: script:l1mdt: stop of /etc/init.d/l1mdt failed (returned 1) Jun 18 00:21:35 sklusp01a clurgmgrd[9379]: stop on script "l1mdt" returned 1 (generic error) Jun 18 00:21:35 sklusp01a clurgmgrd[9379]: Member 2 shutting down Jun 18 00:21:58 sklusp01a clurgmgrd[9379]: Starting disabled service service:l1mdt Jun 18 00:22:14 sklusp01a openais[8586]: [TOTEM] The token was lost in the OPERATIONAL state. Jun 18 00:22:14 sklusp01a openais[8586]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Jun 18 00:22:14 sklusp01a openais[8586]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Jun 18 00:22:14 sklusp01a openais[8586]: [TOTEM] entering GATHER state from 2. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] entering GATHER state from 0. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] Creating commit token because I am the rep. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] Saving state aru 46 high seq received 46 Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] Storing new sequence id for ring 13c Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] entering COMMIT state. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] entering RECOVERY state. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] position [0] member 10.214.21.54: Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] previous ring seq 312 rep 10.214.21.54 Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] aru 46 high delivered 46 received flag 1 Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] Did not need to originate any messages in recovery. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] Sending initial ORF token Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.182) Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:23:14 sklusp01a openais[8586]: [SYNC ] This node is within the primary component and will provide service. Jun 18 00:23:14 sklusp01a openais[8586]: [TOTEM] entering OPERATIONAL state. Jun 18 00:23:14 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.54 Jun 18 00:23:14 sklusp01a openais[8586]: [CPG ] got joinlist message from node 1 Jun 18 00:23:14 sklusp01a kernel: dlm: closing connection to node 2 Jun 18 00:23:14 sklusp01a kernel: Lustre: OBD class driver, http://www.lustre.org/ Jun 18 00:23:14 sklusp01a kernel: Lustre: Lustre Version: 1.8.5 Jun 18 00:23:14 sklusp01a kernel: Lustre: Build Version: 1.8.5-20101116203234-PRISTINE-2.6.18-194.17.1.el5_lustre.1.8.5 Jun 18 00:23:15 sklusp01a kernel: Lustre: Added LNI 10.214.127.54@tcp [8/256/0/180] Jun 18 00:23:15 sklusp01a kernel: Lustre: Accept secure, port 988 Jun 18 00:23:15 sklusp01a kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Jun 18 00:23:15 sklusp01a kernel: init dynlocks cache Jun 18 00:23:15 sklusp01a kernel: ldiskfs created from ext3-2.6-rhel5 Jun 18 00:23:15 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 00:23:15 sklusp01a kernel: LDISKFS-fs warning: mounting fs with errors, running e2fsck is recommended Jun 18 00:23:15 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 00:23:15 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 00:23:15 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 00:23:15 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 00:23:15 sklusp01a kernel: LDISKFS-fs warning: mounting fs with errors, running e2fsck is recommended Jun 18 00:23:15 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 00:23:15 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 00:23:15 sklusp01a kernel: Lustre: MGS MGS started Jun 18 00:23:15 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 00:23:15 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 00:23:15 sklusp01a kernel: Lustre: Enabling ACL Jun 18 00:23:15 sklusp01a kernel: Lustre: 10565:0:(mds_fs.c:677:mds_init_server_data()) RECOVERY: service l1-MDT0000, 54 recoverable clients, 1 delayed clients, last_transno 137438953472 Jun 18 00:23:15 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 00:23:15 sklusp01a kernel: Lustre: l1-MDT0000: Will be in recovery for at least 5:00, or until 54 clients reconnect Jun 18 00:23:15 sklusp01a kernel: Lustre: 10565:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 00:23:15 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 00:23:15 sklusp01a kernel: Lustre: 10565:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 00:23:15 sklusp01a kernel: Lustre: 10565:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0001_UUID Jun 18 00:23:15 sklusp01a kernel: Lustre: 10348:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0001_UUID Jun 18 00:23:15 sklusp01a kernel: Lustre: 10348:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0003_UUID Jun 18 00:23:16 sklusp01a clurgmgrd[9379]: Service service:l1mdt started Jun 18 00:23:20 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264913927 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 5s ago has timed out (5s prior to deadline). Jun 18 00:23:20 sklusp01a kernel: req@ffff8111dd616000 x1405062264913927/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339971800 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 00:23:20 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264913929 sent from l1-OST0002-osc to NID 10.214.127.56@tcp 5s ago has timed out (5s prior to deadline). Jun 18 00:23:20 sklusp01a kernel: req@ffff8111dd6d7000 x1405062264913929/t0 o8->l1-OST0002_UUID@10.214.127.56@tcp:28/4 lens 368/584 e 0 to 1 dl 1339971800 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 00:23:21 sklusp01a kernel: Lustre: 10348:0:(mds_lov.c:1191:mds_notify()) MDS l1-MDT0000: in recovery, not resetting orphans on l1-OST0000_UUID Jun 18 00:23:21 sklusp01a kernel: Lustre: 10348:0:(mds_lov.c:1191:mds_notify()) Skipped 1 previous similar message Jun 18 00:23:42 sklusp01a kernel: Lustre: 10527:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 53 recoverable clients remain Jun 18 00:23:54 sklusp01a kernel: Lustre: 10517:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 52 recoverable clients remain Jun 18 00:24:37 sklusp01a kernel: LustreError: 10518:0:(ldlm_lib.c:944:target_handle_connect()) l1-MDT0000: denying connection for new client 10.214.127.88@tcp (c67b1cbf-4ed5-dafb-323a-0164138a6efb): 51 clients in recovery for 245s Jun 18 00:24:37 sklusp01a kernel: LustreError: 10518:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81121fa1d800 x1405018450167747/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339971977 ref 1 fl Interpret:/0/0 rc -16/0 Jun 18 00:24:42 sklusp01a kernel: Lustre: 10518:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 50 recoverable clients remain Jun 18 00:24:42 sklusp01a kernel: Lustre: 10518:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 1 previous similar message Jun 18 00:24:44 sklusp01a kernel: Lustre: 10528:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 48 recoverable clients remain Jun 18 00:24:44 sklusp01a kernel: Lustre: 10528:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 1 previous similar message Jun 18 00:24:47 sklusp01a kernel: Lustre: 10529:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 43 recoverable clients remain Jun 18 00:24:47 sklusp01a kernel: Lustre: 10529:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 4 previous similar messages Jun 18 00:24:48 sklusp01a kernel: LustreError: 10510:0:(ldlm_lib.c:944:target_handle_connect()) l1-MDT0000: denying connection for new client 10.214.127.216@tcp (f2a81f57-9ad8-2e38-ab0e-74a5c7691316): 43 clients in recovery for 233s Jun 18 00:24:48 sklusp01a kernel: LustreError: 10510:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8111f76dcc00 x1402389213235338/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339971988 ref 1 fl Interpret:/0/0 rc -16/0 Jun 18 00:24:53 sklusp01a kernel: Lustre: 10525:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 28 recoverable clients remain Jun 18 00:24:53 sklusp01a kernel: Lustre: 10525:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 14 previous similar messages Jun 18 00:25:01 sklusp01a kernel: Lustre: 10517:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) l1-MDT0000: 1 recoverable clients remain Jun 18 00:25:01 sklusp01a kernel: Lustre: 10517:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 26 previous similar messages Jun 18 00:25:03 sklusp01a kernel: LDISKFS-fs error (device dm-14): ldiskfs_lookup: unlinked inode 27720411 in dir #29287441 Jun 18 00:25:03 sklusp01a kernel: Remounting filesystem read-only Jun 18 00:25:03 sklusp01a kernel: Lustre: l1-MDT0000: Post recovery failed, rc -2 Jun 18 00:25:03 sklusp01a kernel: Lustre: l1-MDT0000: Recovery period over after 1:21, of 55 clients 54 recovered and 0 were evicted. Jun 18 00:25:03 sklusp01a kernel: Lustre: l1-MDT0000: sending delayed replies to recovered clients Jun 18 00:25:03 sklusp01a kernel: LustreError: 10516:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:03 sklusp01a kernel: LustreError: 10507:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:03 sklusp01a kernel: LustreError: 10507:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:25:03 sklusp01a kernel: LustreError: 10516:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:25:03 sklusp01a kernel: LustreError: 10516:0:(mds_reint.c:251:mds_finish_transno()) Skipped 232 previous similar messages Jun 18 00:25:04 sklusp01a kernel: LustreError: 10526:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:04 sklusp01a kernel: LustreError: 10526:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 4338 previous similar messages Jun 18 00:25:04 sklusp01a kernel: LustreError: 10508:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:25:04 sklusp01a kernel: LustreError: 10508:0:(mds_reint.c:251:mds_finish_transno()) Skipped 3906 previous similar messages Jun 18 00:25:04 sklusp01a kernel: LDISKFS-fs warning (device dm-14): kmmpd: kmmpd being stopped since filesystem has been remounted as readonly. Jun 18 00:25:08 sklusp01a kernel: LustreError: 10502:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:08 sklusp01a kernel: LustreError: 10502:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 55 previous similar messages Jun 18 00:25:08 sklusp01a kernel: LustreError: 10502:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:08 sklusp01a kernel: LustreError: 10502:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111e9106c00 x1405018450167755/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972008 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:25:19 sklusp01a kernel: LustreError: 10523:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:19 sklusp01a kernel: LustreError: 10523:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:19 sklusp01a kernel: LustreError: 10523:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111de9af050 x1402389213235368/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972019 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:25:22 sklusp01a kernel: LustreError: 10506:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:22 sklusp01a kernel: LustreError: 10506:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111f775bc00 x1405018450167764/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972022 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:25:33 sklusp01a kernel: LustreError: 10523:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:33 sklusp01a kernel: LustreError: 10523:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1 previous similar message Jun 18 00:25:33 sklusp01a kernel: LustreError: 10523:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:33 sklusp01a kernel: LustreError: 10523:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111dd5c3800 x1402389213235386/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972033 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:25:38 sklusp01a kernel: LustreError: 10498:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:49 sklusp01a kernel: LustreError: 10514:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:25:49 sklusp01a kernel: LustreError: 10514:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1 previous similar message Jun 18 00:25:49 sklusp01a kernel: LustreError: 10514:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:25:49 sklusp01a kernel: LustreError: 10514:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111de079050 x1402389213235400/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972049 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:25:49 sklusp01a kernel: LustreError: 10514:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 18 00:26:01 sklusp01a kernel: LustreError: 10514:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:26:01 sklusp01a kernel: LustreError: 10514:0:(mds_reint.c:251:mds_finish_transno()) Skipped 48 previous similar messages Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 151 previous similar messages Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) Skipped 1 previous similar message Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111e20bec00 x1402389213235421/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972067 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:26:07 sklusp01a kernel: LustreError: 10516:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 18 00:26:27 sklusp01a kernel: LustreError: 10526:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:26:27 sklusp01a kernel: LustreError: 10526:0:(mds_fs.c:236:mds_client_add()) Skipped 1 previous similar message Jun 18 00:26:49 sklusp01a kernel: LustreError: 10524:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:26:49 sklusp01a kernel: LustreError: 10524:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 3 previous similar messages Jun 18 00:26:49 sklusp01a kernel: LustreError: 10524:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111cdb76800 x1402389213235469/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972109 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:26:49 sklusp01a kernel: LustreError: 10524:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] entering GATHER state from 11. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] Creating commit token because I am the rep. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] Saving state aru 14 high seq received 14 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] Storing new sequence id for ring 140 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] entering COMMIT state. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] entering RECOVERY state. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] position [0] member 10.214.21.54: Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] previous ring seq 316 rep 10.214.21.54 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] aru 14 high delivered 14 received flag 1 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] position [1] member 10.214.21.182: Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] previous ring seq 316 rep 10.214.21.182 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] aru c high delivered c received flag 1 Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] Did not need to originate any messages in recovery. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] Sending initial ORF token Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] CLM CONFIGURATION CHANGE Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] New Configuration: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.54) Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.182) Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] Members Left: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] Members Joined: Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] r(0) ip(10.214.21.182) Jun 18 00:26:52 sklusp01a openais[8586]: [SYNC ] This node is within the primary component and will provide service. Jun 18 00:26:52 sklusp01a openais[8586]: [TOTEM] entering OPERATIONAL state. Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.54 Jun 18 00:26:52 sklusp01a openais[8586]: [CLM ] got nodejoin message 10.214.21.182 Jun 18 00:26:52 sklusp01a openais[8586]: [CPG ] got joinlist message from node 1 Jun 18 00:27:01 sklusp01a kernel: LustreError: 10508:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:27:01 sklusp01a kernel: LustreError: 10508:0:(mds_reint.c:251:mds_finish_transno()) Skipped 138 previous similar messages Jun 18 00:27:02 sklusp01a kernel: LustreError: 10523:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:27:02 sklusp01a kernel: LustreError: 10523:0:(mds_fs.c:236:mds_client_add()) Skipped 2 previous similar messages Jun 18 00:27:20 sklusp01a kernel: dlm: got connection from 2 Jun 18 00:27:56 sklusp01a kernel: LustreError: 10507:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:27:56 sklusp01a kernel: LustreError: 10507:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 154 previous similar messages Jun 18 00:27:56 sklusp01a kernel: LustreError: 10507:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111f63eac00 x1405018450167820/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972176 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:27:56 sklusp01a kernel: LustreError: 10507:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 18 00:28:01 sklusp01a kernel: LustreError: 10524:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:28:01 sklusp01a kernel: LustreError: 10524:0:(mds_reint.c:251:mds_finish_transno()) Skipped 157 previous similar messages Jun 18 00:28:07 sklusp01a kernel: LustreError: 10524:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:28:07 sklusp01a kernel: LustreError: 10524:0:(mds_fs.c:236:mds_client_add()) Skipped 4 previous similar messages Jun 18 00:29:01 sklusp01a kernel: LustreError: 10509:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:29:01 sklusp01a kernel: LustreError: 10509:0:(mds_reint.c:251:mds_finish_transno()) Skipped 133 previous similar messages Jun 18 00:30:01 sklusp01a kernel: LustreError: 10523:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:30:01 sklusp01a kernel: LustreError: 10523:0:(mds_reint.c:251:mds_finish_transno()) Skipped 149 previous similar messages Jun 18 00:30:08 sklusp01a kernel: LustreError: 10502:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:30:08 sklusp01a kernel: LustreError: 10502:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 457 previous similar messages Jun 18 00:30:08 sklusp01a kernel: LustreError: 10502:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111de9aec50 x1405018450167863/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972308 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:30:08 sklusp01a kernel: LustreError: 10502:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Jun 18 00:30:19 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:30:19 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) Skipped 7 previous similar messages Jun 18 00:31:01 sklusp01a kernel: LustreError: 11058:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:31:01 sklusp01a kernel: LustreError: 11058:0:(mds_reint.c:251:mds_finish_transno()) Skipped 155 previous similar messages Jun 18 00:33:01 sklusp01a kernel: LustreError: 10521:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:33:01 sklusp01a kernel: LustreError: 10521:0:(mds_reint.c:251:mds_finish_transno()) Skipped 293 previous similar messages Jun 18 00:34:26 sklusp01a kernel: LustreError: 10501:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:34:26 sklusp01a kernel: LustreError: 10501:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 611 previous similar messages Jun 18 00:34:26 sklusp01a kernel: LustreError: 10501:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111e01dd450 x1405018450167946/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339972566 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:34:26 sklusp01a kernel: LustreError: 10501:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 11 previous similar messages Jun 18 00:34:37 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:34:37 sklusp01a kernel: LustreError: 10516:0:(mds_fs.c:236:mds_client_add()) Skipped 11 previous similar messages Jun 18 00:36:01 sklusp01a kernel: LustreError: 10520:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:36:01 sklusp01a kernel: LustreError: 10520:0:(mds_reint.c:251:mds_finish_transno()) Skipped 449 previous similar messages Jun 18 00:41:01 sklusp01a kernel: LustreError: 10517:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:41:01 sklusp01a kernel: LustreError: 10517:0:(mds_reint.c:251:mds_finish_transno()) Skipped 743 previous similar messages Jun 18 00:43:01 sklusp01a kernel: LustreError: 10500:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:43:01 sklusp01a kernel: LustreError: 10500:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1225 previous similar messages Jun 18 00:43:02 sklusp01a kernel: LustreError: 10523:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111de4cf450 x1405018450168112/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339973082 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:43:02 sklusp01a kernel: LustreError: 10523:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17 previous similar messages Jun 18 00:43:13 sklusp01a kernel: LustreError: 10524:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:43:13 sklusp01a kernel: LustreError: 10524:0:(mds_fs.c:236:mds_client_add()) Skipped 17 previous similar messages Jun 18 00:50:01 sklusp01a kernel: LustreError: 10513:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 00:50:01 sklusp01a kernel: LustreError: 10513:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1351 previous similar messages Jun 18 00:53:01 sklusp01a kernel: LustreError: 10523:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 00:53:01 sklusp01a kernel: LustreError: 10523:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1522 previous similar messages Jun 18 00:53:11 sklusp01a kernel: LustreError: 11057:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111dd5e5000 x1402389213237090/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339973691 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 00:53:11 sklusp01a kernel: LustreError: 11057:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 30 previous similar messages Jun 18 00:53:36 sklusp01a kernel: LustreError: 10512:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 00:53:36 sklusp01a kernel: LustreError: 10512:0:(mds_fs.c:236:mds_client_add()) Skipped 30 previous similar messages Jun 18 01:00:01 sklusp01a kernel: LustreError: 10498:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:00:01 sklusp01a kernel: LustreError: 10498:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1495 previous similar messages Jun 18 01:03:01 sklusp01a kernel: LustreError: 11059:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:03:01 sklusp01a kernel: LustreError: 11059:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1578 previous similar messages Jun 18 01:03:24 sklusp01a kernel: LustreError: 10516:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111f63de400 x1405018450168519/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339974304 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 01:03:24 sklusp01a kernel: LustreError: 10516:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26 previous similar messages Jun 18 01:03:41 sklusp01a kernel: LustreError: 10526:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 01:03:41 sklusp01a kernel: LustreError: 10526:0:(mds_fs.c:236:mds_client_add()) Skipped 26 previous similar messages Jun 18 01:10:01 sklusp01a kernel: LustreError: 10502:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:10:01 sklusp01a kernel: LustreError: 10502:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1565 previous similar messages Jun 18 01:13:01 sklusp01a kernel: LustreError: 10508:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:13:01 sklusp01a kernel: LustreError: 10508:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1538 previous similar messages Jun 18 01:13:36 sklusp01a kernel: LustreError: 10509:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111de6acc50 x1405018450168739/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339974916 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 01:13:36 sklusp01a kernel: LustreError: 10509:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20 previous similar messages Jun 18 01:13:54 sklusp01a kernel: LustreError: 10505:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 01:13:54 sklusp01a kernel: LustreError: 10505:0:(mds_fs.c:236:mds_client_add()) Skipped 20 previous similar messages Jun 18 01:20:01 sklusp01a kernel: LustreError: 10521:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:20:01 sklusp01a kernel: LustreError: 10521:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1527 previous similar messages Jun 18 01:23:01 sklusp01a kernel: LustreError: 10509:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:23:01 sklusp01a kernel: LustreError: 10509:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1564 previous similar messages Jun 18 01:24:05 sklusp01a kernel: LustreError: 10499:0:(mds_fs.c:236:mds_client_add()) unable to start transaction: rc -30 Jun 18 01:24:05 sklusp01a kernel: LustreError: 10499:0:(mds_fs.c:236:mds_client_add()) Skipped 34 previous similar messages Jun 18 01:24:05 sklusp01a kernel: LustreError: 10499:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111e42c1800 x1405018450168972/t0 o38->@:0/0 lens 368/264 e 0 to 0 dl 1339975545 ref 1 fl Interpret:/0/0 rc -30/0 Jun 18 01:24:05 sklusp01a kernel: LustreError: 10499:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages Jun 18 01:30:01 sklusp01a kernel: LustreError: 10516:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:30:01 sklusp01a kernel: LustreError: 10516:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1505 previous similar messages Jun 18 01:33:01 sklusp01a kernel: LustreError: 10512:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:33:01 sklusp01a kernel: LustreError: 10512:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1513 previous similar messages Jun 18 01:35:08 sklusp01a kernel: Lustre: MGS: haven't heard from client 92df43f7-e6f1-4f1e-9e20-89ced1306e12 (at 10.214.127.216@tcp) in 227 seconds. I think it's dead, and I am evicting it. Jun 18 01:35:08 sklusp01a kernel: Lustre: MGS: haven't heard from client 6e372a06-29bc-979c-abf5-e4497b318d54 (at 10.214.127.88@tcp) in 189 seconds. I think it's dead, and I am evicting it. Jun 18 01:36:20 sklusp01a kernel: LustreError: 10483:0:(llog_lvfs.c:577:llog_filp_open()) logfile creation CONFIGS/l1-client: -30 Jun 18 01:36:20 sklusp01a kernel: LustreError: 10483:0:(mgs_handler.c:672:mgs_handle()) MGS handle cmd=501 rc=-30 Jun 18 01:36:20 sklusp01a kernel: LustreError: 10483:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8111dee3fc50 x1405066861871107/t0 o501->a1ca2dd5-7d58-e541-2500-c4bb2d69c598@NET_0x200000ad67fd8_UUID:0/0 lens 264/0 e 0 to 0 dl 1339976186 ref 1 fl Interpret:/0/0 rc 0/0 Jun 18 01:36:20 sklusp01a kernel: LustreError: 10483:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17 previous similar messages Jun 18 01:36:20 sklusp01a kernel: LustreError: 10482:0:(llog_lvfs.c:577:llog_filp_open()) logfile creation CONFIGS/l1-client: -30 Jun 18 01:36:20 sklusp01a kernel: LustreError: 10482:0:(mgs_handler.c:672:mgs_handle()) MGS handle cmd=501 rc=-30 Jun 18 01:40:01 sklusp01a kernel: LustreError: 10510:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:40:01 sklusp01a kernel: LustreError: 10510:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1503 previous similar messages Jun 18 01:41:29 sklusp01a kernel: LustreError: 10482:0:(llog_lvfs.c:577:llog_filp_open()) logfile creation CONFIGS/l1-client: -30 Jun 18 01:41:29 sklusp01a kernel: LustreError: 10482:0:(mgs_handler.c:672:mgs_handle()) MGS handle cmd=501 rc=-30 Jun 18 01:43:01 sklusp01a kernel: LustreError: 10527:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:43:01 sklusp01a kernel: LustreError: 10527:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1502 previous similar messages Jun 18 01:50:01 sklusp01a kernel: LustreError: 10528:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 01:50:01 sklusp01a kernel: LustreError: 10528:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1504 previous similar messages Jun 18 01:53:01 sklusp01a kernel: LustreError: 11056:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 01:53:01 sklusp01a kernel: LustreError: 11056:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1501 previous similar messages Jun 18 02:00:01 sklusp01a kernel: LustreError: 10514:0:(mds_reint.c:251:mds_finish_transno()) fsfilt_start: -30 Jun 18 02:00:01 sklusp01a kernel: LustreError: 10514:0:(mds_reint.c:251:mds_finish_transno()) Skipped 1494 previous similar messages Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 5d65abcd-e65b-5eda-09aa-5a209d0b2d1b at idx 28/11776 in last_rcvd sync rc -30 Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 4f4b28fe-c0b8-7917-086b-d46c154bcef5 at idx 13/9856 in last_rcvd sync rc -30 Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 41d0fdfb-de96-873a-f312-692a7b82479a at idx 43/13696 in last_rcvd sync rc -30 Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 1 previous similar message Jun 18 02:01:38 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client de039a49-b7ee-d871-362e-ff466b801f2f at idx 19/10624 in last_rcvd sync rc -30 Jun 18 02:01:40 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:40 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client d61d19b0-b4c3-9b01-7f54-5f7316e485f3 at idx 38/13056 in last_rcvd sync rc -30 Jun 18 02:01:40 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client c490dce9-2cbf-19ca-51de-101fce19a4b9 at idx 33/12416 in last_rcvd sync rc -30 Jun 18 02:01:41 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 24d4883c-ee48-7496-610e-b631f116fc9a at idx 1/8320 in last_rcvd sync rc -30 Jun 18 02:01:41 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 4311313b-a176-48df-4195-7422510275bf at idx 14/9984 in last_rcvd sync rc -30 Jun 18 02:01:42 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:42 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 3 previous similar messages Jun 18 02:01:42 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 3665dba5-f67f-e84e-0771-ff99f9b6aeda at idx 41/13440 in last_rcvd sync rc -30 Jun 18 02:01:42 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client a7695391-fb4a-dd54-8db0-687f47f8b8c2 at idx 36/12800 in last_rcvd sync rc -30 Jun 18 02:01:43 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 3f1edaa3-0210-1a17-e42e-3a17ededa31d at idx 49/14464 in last_rcvd sync rc -30 Jun 18 02:01:44 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 22361259-5bed-3a6b-6d5b-370778c514bb at idx 15/10112 in last_rcvd sync rc -30 Jun 18 02:01:44 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 625f1d15-7c67-67ca-b2a7-51f89eb4122b at idx 52/14848 in last_rcvd sync rc -30 Jun 18 02:01:45 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 016393d9-bc38-e128-1e0a-92411375ba50 at idx 20/10752 in last_rcvd sync rc -30 Jun 18 02:01:45 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 4933b0b0-e00d-1ff0-3f9e-5f36eff12fc3 at idx 55/15232 in last_rcvd sync rc -30 Jun 18 02:01:45 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client fd9d4d6d-e4b0-e440-f0e5-929293425d53 at idx 50/14592 in last_rcvd sync rc -30 Jun 18 02:01:46 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 1e9b3dc0-e38d-7d58-1523-d61e478553c6 at idx 54/15104 in last_rcvd sync rc -30 Jun 18 02:01:46 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:46 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 8 previous similar messages Jun 18 02:01:46 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 4d5622fd-1864-2a54-2eba-6bdc041659f3 at idx 17/10368 in last_rcvd sync rc -30 Jun 18 02:01:46 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 59423f6d-5a8f-5a85-5071-14f24ce6d120 at idx 37/12928 in last_rcvd sync rc -30 Jun 18 02:01:47 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client b61f31ec-47ca-04ba-a950-72f0305bd159 at idx 30/12032 in last_rcvd sync rc -30 Jun 18 02:01:47 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 3400cf27-e92f-8617-3f85-17e7587c3d97 at idx 18/10496 in last_rcvd sync rc -30 Jun 18 02:01:48 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client c569d229-8525-fdf5-0f51-922ee9267f38 at idx 51/14720 in last_rcvd sync rc -30 Jun 18 02:01:48 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 8764c895-2c11-8bec-470d-d63091c329de at idx 16/10240 in last_rcvd sync rc -30 Jun 18 02:01:49 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 19116079-452d-fc3d-d594-4bb9c6607bdb at idx 42/13568 in last_rcvd sync rc -30 Jun 18 02:01:49 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 694c7256-d458-bdc6-4cb0-8008ad3f8164 at idx 21/10880 in last_rcvd sync rc -30 Jun 18 02:01:49 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client d06ac76f-2613-bbce-f338-662ca8748808 at idx 5/8832 in last_rcvd sync rc -30 Jun 18 02:01:49 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client c05d628b-56b6-57c5-0b53-5b815d896cc4 at idx 7/9088 in last_rcvd sync rc -30 Jun 18 02:01:49 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 8147081a-2074-03a6-ce8c-7afe77da5275 at idx 45/13952 in last_rcvd sync rc -30 Jun 18 02:01:50 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 5ec40066-86fd-466e-ca6c-f1ffd414ff3b at idx 23/11136 in last_rcvd sync rc -30 Jun 18 02:01:50 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 309902d1-6516-0a44-18db-a13c3a97615c at idx 48/14336 in last_rcvd sync rc -30 Jun 18 02:01:50 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client a83e77c2-0433-4c28-f0cf-a889b1119c25 at idx 6/8960 in last_rcvd sync rc -30 Jun 18 02:01:50 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client df3bdabf-0e23-accd-6a8a-24127c6a36ef at idx 35/12672 in last_rcvd sync rc -30 Jun 18 02:01:50 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client ccc769a5-6dd6-4e8d-13a3-a966dcc34280 at idx 11/9600 in last_rcvd sync rc -30 Jun 18 02:01:51 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 2175ffb6-7934-5b3c-3d76-099ad4dd2b5e at idx 32/12288 in last_rcvd sync rc -30 Jun 18 02:01:51 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 2b40bc66-76ee-5a11-d97c-68f27b400fd1 at idx 24/11264 in last_rcvd sync rc -30 Jun 18 02:01:51 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client be32abee-74f8-915f-0909-fd18477b1966 at idx 39/13184 in last_rcvd sync rc -30 Jun 18 02:01:51 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 6d86d4d1-09c5-116f-aad1-6a64b9186fb4 at idx 3/8576 in last_rcvd sync rc -30 Jun 18 02:01:51 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 98eeda85-b21f-fb67-d39d-5cb1025cc3fd at idx 53/14976 in last_rcvd sync rc -30 Jun 18 02:01:52 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 71ec4a8c-cd4b-4526-dbef-5759693ef4ba at idx 12/9728 in last_rcvd sync rc -30 Jun 18 02:01:52 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client e05ba413-512c-5877-d175-6d8f0783f4cd at idx 4/8704 in last_rcvd sync rc -30 Jun 18 02:01:52 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 02c9c83f-870c-a11b-eb45-795c5bc92317 at idx 26/11520 in last_rcvd sync rc -30 Jun 18 02:01:52 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 76353444-d986-3906-cf37-02028e59ffd5 at idx 8/9216 in last_rcvd sync rc -30 Jun 18 02:01:53 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 82f2579d-1362-ce5c-034f-1e780bee624b at idx 27/11648 in last_rcvd sync rc -30 Jun 18 02:01:53 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 99351880-01fa-d873-9559-acf5a17ce62d at idx 44/13824 in last_rcvd sync rc -30 Jun 18 02:01:53 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 54cec4e3-7583-ebca-fce6-4ce6148ceb1e at idx 25/11392 in last_rcvd sync rc -30 Jun 18 02:01:54 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client f7a5352b-1583-939a-b41b-d720d8a3f582 at idx 46/14080 in last_rcvd sync rc -30 Jun 18 02:01:54 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:01:54 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 28 previous similar messages Jun 18 02:01:54 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client d1cd7a9b-498f-3880-5138-0c5d6ba835a5 at idx 34/12544 in last_rcvd sync rc -30 Jun 18 02:01:55 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 8e633c45-babf-60ac-5445-0aef319f613e at idx 22/11008 in last_rcvd sync rc -30 Jun 18 02:01:55 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client ac1973d7-cbe0-487b-522f-53a1ba02f960 at idx 0/8192 in last_rcvd sync rc -30 Jun 18 02:06:13 sklusp01a kernel: LustreError: 10521:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 Jun 18 02:06:13 sklusp01a kernel: LustreError: 10521:0:(fsfilt-ldiskfs.c:366:fsfilt_ldiskfs_start()) Skipped 1556 previous similar messages Jun 18 02:06:15 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:06:15 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 2 previous similar messages Jun 18 02:06:15 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 37a303a7-034b-7047-0831-a44a50224a38 at idx 40/13312 in last_rcvd sync rc -30 Jun 18 02:06:42 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client a2d09d17-5eeb-04a7-e22b-b176d36265d2 at idx 29/11904 in last_rcvd sync rc -30 Jun 18 02:07:30 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:07:30 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) Skipped 1 previous similar message Jun 18 02:07:30 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 851872de-cd27-e0f1-3e50-108dde93ddda at idx 47/14208 in last_rcvd sync rc -30 Jun 18 02:09:20 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes) Jun 18 02:09:20 sklusp01a kernel: LustreError: 10307:0:(mds_fs.c:343:mds_client_free()) zero out client 25bfec40-1763-2bba-28e0-d09099b16a28 at idx 2/8448 in last_rcvd sync rc -30 Jun 18 02:16:09 sklusp01a clurgmgrd[9379]: Stopping service service:l1mdt Jun 18 02:16:09 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 18 02:16:09 sklusp01a kernel: LustreError: 12690:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 18 02:16:09 sklusp01a kernel: LustreError: 12690:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 18 02:16:09 sklusp01a kernel: Lustre: Failing over l1-OST0005-osc Jun 18 02:16:09 sklusp01a kernel: Lustre: MGS has stopped. Jun 18 02:16:09 sklusp01a kernel: LustreError: 10307:0:(fsfilt-ldiskfs.c:1332:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (512 bytes) Jun 18 02:16:09 sklusp01a kernel: LustreError: 10307:0:(handler.c:1871:mds_update_server_data()) error writing MDS server data: rc = -30 Jun 18 02:16:09 sklusp01a kernel: Lustre: l1-MDT0000: shutting down for failover; client state will be preserved. Jun 18 02:16:09 sklusp01a kernel: Lustre: MDT l1-MDT0000 has stopped. Jun 18 02:16:12 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 02:16:12 sklusp01a kernel: Lustre: server umount l1-MDT0000 complete Jun 18 02:16:12 sklusp01a clurgmgrd[9379]: Service service:l1mdt is disabled Jun 18 02:17:28 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 02:17:28 sklusp01a kernel: LDISKFS-fs warning: mounting fs with errors, running e2fsck is recommended Jun 18 02:17:28 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 02:17:28 sklusp01a kernel: LDISKFS-fs: recovery complete. Jun 18 02:17:28 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 02:20:51 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:00:23 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:00:23 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:00:23 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:02:50 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:34:16 sklusp01a clurgmgrd[9379]: Starting disabled service service:l1mdt Jun 18 03:34:16 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:34:16 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:34:16 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:34:16 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:34:16 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:34:16 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:34:16 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:34:16 sklusp01a kernel: Lustre: MGS MGS started Jun 18 03:34:16 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 03:34:16 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 03:34:16 sklusp01a kernel: Lustre: Enabling ACL Jun 18 03:34:16 sklusp01a kernel: Lustre: 1362:0:(mds_fs.c:677:mds_init_server_data()) RECOVERY: service l1-MDT0000, 54 recoverable clients, 0 delayed clients, last_transno 141733920768 Jun 18 03:34:16 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 03:34:16 sklusp01a kernel: Lustre: l1-MDT0000: Will be in recovery for at least 5:00, or until 54 clients reconnect Jun 18 03:34:16 sklusp01a kernel: Lustre: 1362:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 03:34:16 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 03:34:16 sklusp01a kernel: Lustre: 1362:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 03:34:16 sklusp01a kernel: Lustre: 1362:0:(mds_lov.c:1155:mds_notify()) Skipped 4 previous similar messages Jun 18 03:34:16 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915848 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 0s ago has failed due to network error (5s prior to deadline). Jun 18 03:34:16 sklusp01a kernel: req@ffff8111cdd04c00 x1405062264915848/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983261 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:16 sklusp01a kernel: Lustre: 1362:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0001_UUID Jun 18 03:34:16 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 18 03:34:16 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915850 sent from l1-OST0002-osc to NID 10.214.127.56@tcp 0s ago has failed due to network error (5s prior to deadline). Jun 18 03:34:16 sklusp01a kernel: req@ffff811199f63c00 x1405062264915850/t0 o8->l1-OST0002_UUID@10.214.127.56@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983261 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:16 sklusp01a clurgmgrd[9379]: Service service:l1mdt started Jun 18 03:34:21 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915849 sent from l1-OST0001-osc to NID 10.214.127.183@tcp 5s ago has timed out (5s prior to deadline). Jun 18 03:34:21 sklusp01a kernel: req@ffff8112060bd800 x1405062264915849/t0 o8->l1-OST0001_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983261 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:21 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 18 03:34:23 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915857 sent from l1-OST0001-osc to NID 10.214.127.55@tcp 0s ago has failed due to network error (5s prior to deadline). Jun 18 03:34:23 sklusp01a kernel: req@ffff81121dc8f400 x1405062264915857/t0 o8->l1-OST0001_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983268 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:23 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 18 03:34:28 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915856 sent from l1-OST0000-osc to NID 10.214.127.183@tcp 5s ago has timed out (5s prior to deadline). Jun 18 03:34:28 sklusp01a kernel: req@ffff81121dc8f000 x1405062264915856/t0 o8->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983268 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:28 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 18 03:34:29 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 1s Jun 18 03:34:29 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0001-osc: tried all connections, increasing latency to 1s Jun 18 03:34:35 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915863 sent from l1-OST0001-osc to NID 10.214.127.183@tcp 6s ago has timed out (6s prior to deadline). Jun 18 03:34:35 sklusp01a kernel: req@ffff811049642000 x1405062264915863/t0 o8->l1-OST0001_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983275 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:35 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 18 03:34:48 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915870 sent from l1-OST0001-osc to NID 10.214.127.55@tcp 0s ago has failed due to network error (6s prior to deadline). Jun 18 03:34:48 sklusp01a kernel: req@ffff81103cc80800 x1405062264915870/t0 o8->l1-OST0001_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983294 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:34:48 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 18 03:34:55 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 2s Jun 18 03:34:55 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 4 previous similar messages Jun 18 03:35:13 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915883 sent from l1-OST0001-osc to NID 10.214.127.55@tcp 0s ago has failed due to network error (7s prior to deadline). Jun 18 03:35:13 sklusp01a kernel: req@ffff81121db68800 x1405062264915883/t0 o8->l1-OST0001_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983320 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:35:13 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jun 18 03:35:21 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 3s Jun 18 03:35:21 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 03:35:46 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915895 sent from l1-OST0000-osc to NID 10.214.127.183@tcp 8s ago has timed out (8s prior to deadline). Jun 18 03:35:46 sklusp01a kernel: req@ffff81110a892c00 x1405062264915895/t0 o8->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983346 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:35:46 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jun 18 03:35:47 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 4s Jun 18 03:35:47 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 03:36:06 sklusp01a clurgmgrd[9379]: Stopping service service:l1mdt Jun 18 03:36:06 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 18 03:36:06 sklusp01a kernel: Lustre: Skipped 6 previous similar messages Jun 18 03:36:06 sklusp01a kernel: Lustre: l1-MDT0000: shutting down for failover; client state will be preserved. Jun 18 03:36:06 sklusp01a kernel: Lustre: MDT l1-MDT0000 has stopped. Jun 18 03:36:06 sklusp01a kernel: LustreError: 1978:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 18 03:36:06 sklusp01a kernel: LustreError: 1978:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 18 03:36:06 sklusp01a kernel: Lustre: MGS has stopped. Jun 18 03:36:06 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:36:06 sklusp01a kernel: Lustre: server umount l1-MDT0000 complete Jun 18 03:36:07 sklusp01a clurgmgrd[9379]: Service service:l1mdt is disabled Jun 18 03:37:57 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:37:57 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:37:57 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:37:57 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:37:57 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:37:57 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:37:57 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:37:57 sklusp01a kernel: Lustre: MGS MGS started Jun 18 03:37:57 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 03:37:57 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 03:37:57 sklusp01a kernel: Lustre: Enabling ACL Jun 18 03:37:57 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 03:37:57 sklusp01a kernel: Lustre: 2640:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 03:37:57 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 03:37:57 sklusp01a kernel: Lustre: 2640:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 03:37:57 sklusp01a kernel: Lustre: 2640:0:(mds_lov.c:1155:mds_notify()) Skipped 4 previous similar messages Jun 18 03:37:57 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264915922 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 0s ago has failed due to network error (5s prior to deadline). Jun 18 03:37:57 sklusp01a kernel: req@ffff811178174400 x1405062264915922/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983482 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:37:57 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jun 18 03:37:57 sklusp01a kernel: Lustre: l1-MDT0000: Aborting recovery. Jun 18 03:38:13 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 18 03:38:13 sklusp01a kernel: Lustre: Skipped 7 previous similar messages Jun 18 03:38:13 sklusp01a kernel: Lustre: l1-MDT0000: shutting down for failover; client state will be preserved. Jun 18 03:38:13 sklusp01a kernel: Lustre: MDT l1-MDT0000 has stopped. Jun 18 03:38:13 sklusp01a kernel: LustreError: 2664:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 18 03:38:13 sklusp01a kernel: LustreError: 2664:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 18 03:38:13 sklusp01a kernel: Lustre: MGS has stopped. Jun 18 03:38:13 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:38:13 sklusp01a kernel: Lustre: server umount l1-MDT0000 complete Jun 18 03:38:27 sklusp01a clurgmgrd[9379]: Starting disabled service service:l1mdt Jun 18 03:38:27 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:38:27 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:38:27 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:38:27 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 03:38:27 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 03:38:27 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 03:38:27 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 03:38:27 sklusp01a kernel: Lustre: MGS MGS started Jun 18 03:38:27 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 03:38:27 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 03:38:27 sklusp01a kernel: Lustre: Enabling ACL Jun 18 03:38:27 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 03:38:27 sklusp01a kernel: Lustre: 2879:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 03:38:27 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 03:38:27 sklusp01a kernel: Lustre: 2879:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 03:38:27 sklusp01a kernel: Lustre: 2879:0:(mds_lov.c:1155:mds_notify()) Skipped 5 previous similar messages Jun 18 03:38:27 sklusp01a clurgmgrd[9379]: Service service:l1mdt started Jun 18 03:38:39 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 1s Jun 18 03:38:39 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 03:38:58 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 2s Jun 18 03:38:58 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 03:39:20 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 3s Jun 18 03:39:20 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 03:39:20 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 03:39:20 sklusp01a kernel: Lustre: l1-OST0001-osc: Connection restored to service l1-OST0001 using nid 10.214.127.55@tcp. Jun 18 03:39:20 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 03:39:20 sklusp01a kernel: Lustre: l1-OST0000-osc: Connection restored to service l1-OST0000 using nid 10.214.127.55@tcp. Jun 18 03:39:20 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0000_UUID now active, resetting orphans Jun 18 03:39:20 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0001_UUID now active, resetting orphans Jun 18 03:39:48 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 03:39:48 sklusp01a kernel: Lustre: l1-OST0002-osc: Connection restored to service l1-OST0002 using nid 10.214.127.56@tcp. Jun 18 03:39:48 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0002_UUID now active, resetting orphans Jun 18 03:40:13 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 03:40:13 sklusp01a kernel: Lustre: l1-OST0003-osc: Connection restored to service l1-OST0003 using nid 10.214.127.56@tcp. Jun 18 03:40:13 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0005_UUID now active, resetting orphans Jun 18 03:40:21 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062264916011 sent from l1-OST0004-osc to NID 10.214.127.185@tcp 8s ago has timed out (8s prior to deadline). Jun 18 03:40:21 sklusp01a kernel: req@ffff81115d33f000 x1405062264916011/t0 o8->l1-OST0004_UUID@10.214.127.185@tcp:28/4 lens 368/584 e 0 to 1 dl 1339983621 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 03:40:21 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 46 previous similar messages Jun 18 03:40:22 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0004-osc: tried all connections, increasing latency to 4s Jun 18 03:40:22 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 4 previous similar messages Jun 18 03:40:22 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 03:40:22 sklusp01a kernel: Lustre: 10348:0:(quota_master.c:1716:mds_quota_recovery()) Skipped 1 previous similar message Jun 18 03:40:22 sklusp01a kernel: Lustre: l1-OST0004-osc: Connection restored to service l1-OST0004 using nid 10.214.127.57@tcp. Jun 18 03:40:22 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 03:40:22 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0004_UUID now active, resetting orphans Jun 18 03:40:22 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 05:48:24 sklusp01a kernel: Lustre: 10347:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062265381840 sent from l1-OST0004-osc to NID 10.214.127.57@tcp 17s ago has timed out (17s prior to deadline). Jun 18 05:48:24 sklusp01a kernel: req@ffff8111263c1400 x1405062265381840/t0 o400->l1-OST0004_UUID@10.214.127.57@tcp:28/4 lens 192/384 e 0 to 1 dl 1339991304 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 05:48:24 sklusp01a kernel: Lustre: l1-OST0004-osc: Connection to service l1-OST0004 via nid 10.214.127.57@tcp was lost; in progress operations using this service will wait for recovery to complete. Jun 18 05:48:24 sklusp01a kernel: Lustre: l1-OST0005-osc: Connection to service l1-OST0005 via nid 10.214.127.57@tcp was lost; in progress operations using this service will wait for recovery to complete. Jun 18 05:48:38 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0004-osc: tried all connections, increasing latency to 2s Jun 18 05:48:49 sklusp01a kernel: Lustre: l1-OST0002-osc: Connection to service l1-OST0002 via nid 10.214.127.56@tcp was lost; in progress operations using this service will wait for recovery to complete. Jun 18 05:48:54 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0004-osc: tried all connections, increasing latency to 3s Jun 18 05:48:54 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message Jun 18 05:49:02 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405062265381857 sent from l1-OST0004-osc to NID 10.214.127.57@tcp 8s ago has timed out (8s prior to deadline). Jun 18 05:49:02 sklusp01a kernel: req@ffff81121d835c00 x1405062265381857/t0 o8->l1-OST0004_UUID@10.214.127.57@tcp:28/4 lens 368/584 e 0 to 1 dl 1339991342 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 05:49:02 sklusp01a kernel: Lustre: 10348:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jun 18 05:49:13 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) l1-OST0004-osc: tried all connections, increasing latency to 4s Jun 18 05:49:13 sklusp01a kernel: Lustre: 10349:0:(import.c:517:import_select_connection()) Skipped 3 previous similar messages Jun 18 05:49:14 sklusp01a kernel: Lustre: l1-OST0000-osc: Connection to service l1-OST0000 via nid 10.214.127.55@tcp was lost; in progress operations using this service will wait for recovery to complete. Jun 18 05:49:14 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 05:49:14 sklusp01a clurgmgrd[9379]: Stopping service service:l1mdt Jun 18 05:49:14 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 18 05:49:14 sklusp01a kernel: Lustre: Skipped 7 previous similar messages Jun 18 05:49:14 sklusp01a kernel: LustreError: 8924:0:(mds_fs.c:1078:mds_obd_destroy()) destroying non-existent object 81916411 4e1f1fb:f64bf0e1: rc 0 Jun 18 05:49:14 sklusp01a kernel: LustreError: 8924:0:(llog_obd.c:397:llog_obd_origin_cleanup()) failure destroying log during cleanup: -2 Jun 18 05:49:14 sklusp01a kernel: Lustre: l1-MDT0000: shutting down for failover; client state will be preserved. Jun 18 05:49:15 sklusp01a kernel: Lustre: MDT l1-MDT0000 has stopped. Jun 18 05:49:15 sklusp01a kernel: LustreError: 8924:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 18 05:49:15 sklusp01a kernel: LustreError: 8924:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 18 05:49:15 sklusp01a kernel: Lustre: MGS has stopped. Jun 18 05:49:15 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 05:49:15 sklusp01a kernel: Lustre: server umount l1-MDT0000 complete Jun 18 05:49:15 sklusp01a clurgmgrd[9379]: Service service:l1mdt is disabled Jun 18 05:50:18 sklusp01a kernel: exit dynlocks cache Jun 18 05:50:20 sklusp01a kernel: Lustre: Acceptor stopping Jun 18 05:50:22 sklusp01a kernel: Lustre: Removed LNI 10.214.127.54@tcp Jun 18 05:55:32 sklusp01a kernel: init dynlocks cache Jun 18 05:55:32 sklusp01a kernel: ldiskfs created from ext3-2.6-rhel5 Jun 18 05:55:32 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 05:55:32 sklusp01a kernel: LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended Jun 18 05:55:32 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 05:55:32 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 05:55:32 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 05:57:11 sklusp01a kernel: Lustre: OBD class driver, http://www.lustre.org/ Jun 18 05:57:11 sklusp01a kernel: Lustre: Lustre Version: 1.8.5 Jun 18 05:57:11 sklusp01a kernel: Lustre: Build Version: 1.8.5-20101116203234-PRISTINE-2.6.18-194.17.1.el5_lustre.1.8.5 Jun 18 05:57:11 sklusp01a kernel: Lustre: Added LNI 10.214.127.54@tcp [8/256/0/180] Jun 18 05:57:11 sklusp01a kernel: Lustre: Accept secure, port 988 Jun 18 05:57:11 sklusp01a kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Jun 18 05:57:11 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 05:57:11 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 05:57:11 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 05:57:11 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 05:57:11 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 05:57:11 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 05:57:11 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 05:57:11 sklusp01a kernel: Lustre: MGS MGS started Jun 18 05:57:11 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 05:57:11 sklusp01a kernel: Lustre: MGS: Logs for fs l1 were removed by user request. All servers must be restarted in order to regenerate the logs. Jun 18 05:57:11 sklusp01a kernel: Lustre: Setting parameter l1-MDT0000.mdt.group_upcall in log l1-MDT0000 Jun 18 05:57:11 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 05:57:11 sklusp01a kernel: Lustre: Enabling ACL Jun 18 05:57:11 sklusp01a kernel: Lustre: l1-MDT0000: new disk, initializing Jun 18 05:57:11 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 05:57:11 sklusp01a kernel: Lustre: 11492:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 05:57:11 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 06:02:27 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:02:32 sklusp01a kernel: Lustre: 12804:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 06:02:32 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/1 OSTs are active, abort quota recovery Jun 18 06:02:32 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0000_UUID now active, resetting orphans Jun 18 06:04:23 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:04:26 sklusp01a kernel: Lustre: 13284:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0001_UUID Jun 18 06:04:26 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 1/2 OSTs are active, abort quota recovery Jun 18 06:04:26 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0001_UUID now active, resetting orphans Jun 18 06:06:24 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:06:28 sklusp01a kernel: Lustre: 13762:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0002_UUID Jun 18 06:06:28 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 2/3 OSTs are active, abort quota recovery Jun 18 06:06:28 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0002_UUID now active, resetting orphans Jun 18 06:06:52 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:06:57 sklusp01a kernel: Lustre: 13913:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0003_UUID Jun 18 06:06:57 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 3/4 OSTs are active, abort quota recovery Jun 18 06:06:57 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0003_UUID now active, resetting orphans Jun 18 06:09:35 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:09:41 sklusp01a kernel: Lustre: 14525:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0004_UUID Jun 18 06:09:41 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 4/5 OSTs are active, abort quota recovery Jun 18 06:09:41 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0004_UUID now active, resetting orphans Jun 18 06:10:03 sklusp01a kernel: Lustre: MGS: Regenerating l1-OSTffff log by user request. Jun 18 06:10:08 sklusp01a kernel: Lustre: 14669:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0005_UUID Jun 18 06:10:08 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 5/6 OSTs are active, abort quota recovery Jun 18 06:10:08 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0005_UUID now active, resetting orphans Jun 18 06:32:21 sklusp01a kernel: Lustre: Failing over l1-MDT0000 Jun 18 06:32:21 sklusp01a kernel: Lustre: Failing over l1-OST0005-osc Jun 18 06:32:21 sklusp01a kernel: Lustre: l1-MDT0000: shutting down for failover; client state will be preserved. Jun 18 06:32:21 sklusp01a kernel: Lustre: MDT l1-MDT0000 has stopped. Jun 18 06:32:21 sklusp01a kernel: LustreError: 28926:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 18 06:32:21 sklusp01a kernel: LustreError: 28926:0:(ldlm_request.c:1583:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 18 06:32:21 sklusp01a kernel: Lustre: MGS has stopped. Jun 18 06:32:21 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 06:32:21 sklusp01a kernel: Lustre: server umount l1-MDT0000 complete Jun 18 06:34:04 sklusp01a clurgmgrd[9379]: Starting disabled service service:l1mdt Jun 18 06:34:04 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 06:34:04 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 06:34:04 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 06:34:04 sklusp01a multipathd: dm-14: umount map (uevent) Jun 18 06:34:04 sklusp01a kernel: kjournald starting. Commit interval 5 seconds Jun 18 06:34:04 sklusp01a kernel: LDISKFS FS on dm-14, internal journal Jun 18 06:34:04 sklusp01a kernel: LDISKFS-fs: mounted filesystem with ordered data mode. Jun 18 06:34:04 sklusp01a kernel: Lustre: MGS MGS started Jun 18 06:34:04 sklusp01a kernel: Lustre: MGC10.214.127.54@tcp: Reactivating import Jun 18 06:34:04 sklusp01a kernel: Lustre: Enabling user_xattr Jun 18 06:34:04 sklusp01a kernel: Lustre: Enabling ACL Jun 18 06:34:04 sklusp01a kernel: Lustre: l1-MDT0000: Now serving l1-MDT0000 on /dev/vgl1mdt/lvol1 with recovery enabled Jun 18 06:34:04 sklusp01a kernel: Lustre: 29438:0:(lproc_mds.c:271:lprocfs_wr_group_upcall()) l1-MDT0000: group upcall set to /usr/sbin/l_getgroups Jun 18 06:34:04 sklusp01a kernel: Lustre: l1-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups Jun 18 06:34:04 sklusp01a kernel: Lustre: 29438:0:(mds_lov.c:1155:mds_notify()) MDS l1-MDT0000: add target l1-OST0000_UUID Jun 18 06:34:05 sklusp01a clurgmgrd[9379]: Service service:l1mdt started Jun 18 06:34:09 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183289 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 5s ago has timed out (5s prior to deadline). Jun 18 06:34:09 sklusp01a kernel: req@ffff811040328400 x1405083274183289/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994049 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:09 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183290 sent from l1-OST0001-osc to NID 10.214.127.183@tcp 5s ago has timed out (5s prior to deadline). Jun 18 06:34:09 sklusp01a kernel: req@ffff811210d31c00 x1405083274183290/t0 o8->l1-OST0001_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994049 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:10 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183291 sent from l1-OST0002-osc to NID 10.214.127.56@tcp 5s ago has timed out (5s prior to deadline). Jun 18 06:34:10 sklusp01a kernel: req@ffff811072588400 x1405083274183291/t0 o8->l1-OST0002_UUID@10.214.127.56@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994050 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:27 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183297 sent from l1-OST0000-osc to NID 10.214.127.183@tcp 5s ago has timed out (5s prior to deadline). Jun 18 06:34:27 sklusp01a kernel: req@ffff811072588400 x1405083274183297/t0 o8->l1-OST0000_UUID@10.214.127.183@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994067 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:27 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 18 06:34:27 sklusp01a kernel: Lustre: l1-MDT0000: temporarily refusing client connection from 10.214.127.215@tcp Jun 18 06:34:27 sklusp01a kernel: LustreError: 29368:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-11) req@ffff8111e8d65000 x1405085618798599/t0 o38->@:0/0 lens 368/0 e 0 to 0 dl 1339994167 ref 1 fl Interpret:/0/0 rc -11/0 Jun 18 06:34:28 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 1s Jun 18 06:34:28 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0001-osc: tried all connections, increasing latency to 1s Jun 18 06:34:34 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183303 sent from l1-OST0000-osc to NID 10.214.127.55@tcp 6s ago has timed out (6s prior to deadline). Jun 18 06:34:34 sklusp01a kernel: req@ffff811203fce000 x1405083274183303/t0 o8->l1-OST0000_UUID@10.214.127.55@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994074 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:34 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 18 06:34:47 sklusp01a kernel: LustreError: 11-0: an error occurred while communicating with 10.214.127.183@tcp. The ost_connect operation failed with -19 Jun 18 06:34:47 sklusp01a kernel: LustreError: 11-0: an error occurred while communicating with 10.214.127.55@tcp. The ost_connect operation failed with -19 Jun 18 06:34:47 sklusp01a kernel: Lustre: l1-MDT0000: temporarily refusing client connection from 10.214.127.215@tcp Jun 18 06:34:47 sklusp01a kernel: LustreError: 29369:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-11) req@ffff81116ad4dc00 x1405085618798602/t0 o38->@:0/0 lens 368/0 e 0 to 0 dl 1339994187 ref 1 fl Interpret:/0/0 rc -11/0 Jun 18 06:34:53 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183312 sent from l1-OST0002-osc to NID 10.214.127.184@tcp 6s ago has timed out (6s prior to deadline). Jun 18 06:34:53 sklusp01a kernel: req@ffff8111ca48e400 x1405083274183312/t0 o8->l1-OST0002_UUID@10.214.127.184@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994093 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:34:53 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 18 06:34:54 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0000-osc: tried all connections, increasing latency to 2s Jun 18 06:34:54 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) Skipped 4 previous similar messages Jun 18 06:34:54 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 06:34:54 sklusp01a kernel: Lustre: l1-OST0001-osc: Connection restored to service l1-OST0001 using nid 10.214.127.183@tcp. Jun 18 06:34:54 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0001_UUID now active, resetting orphans Jun 18 06:35:01 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183318 sent from l1-OST0002-osc to NID 10.214.127.56@tcp 7s ago has timed out (7s prior to deadline). Jun 18 06:35:01 sklusp01a kernel: req@ffff8111e113a400 x1405083274183318/t0 o8->l1-OST0002_UUID@10.214.127.56@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994101 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:35:01 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 18 06:35:19 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183333 sent from l1-OST0002-osc to NID 10.214.127.184@tcp 7s ago has timed out (7s prior to deadline). Jun 18 06:35:19 sklusp01a kernel: req@ffff81121f96b000 x1405083274183333/t0 o8->l1-OST0002_UUID@10.214.127.184@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994119 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:35:19 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 18 06:35:20 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0002-osc: tried all connections, increasing latency to 3s Jun 18 06:35:20 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) Skipped 5 previous similar messages Jun 18 06:35:37 sklusp01a kernel: LustreError: 11-0: an error occurred while communicating with 10.214.127.184@tcp. The ost_connect operation failed with -19 Jun 18 06:35:46 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0002-osc: tried all connections, increasing latency to 4s Jun 18 06:35:46 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) Skipped 3 previous similar messages Jun 18 06:35:46 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 06:35:46 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Skipped 1 previous similar message Jun 18 06:35:46 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0003_UUID now active, resetting orphans Jun 18 06:35:46 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 06:35:46 sklusp01a kernel: Lustre: l1-OST0003-osc: Connection restored to service l1-OST0003 using nid 10.214.127.184@tcp. Jun 18 06:35:46 sklusp01a kernel: Lustre: Skipped 1 previous similar message Jun 18 06:35:55 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1405083274183351 sent from l1-OST0005-osc to NID 10.214.127.185@tcp 9s ago has timed out (9s prior to deadline). Jun 18 06:35:55 sklusp01a kernel: req@ffff8112057d2000 x1405083274183351/t0 o8->l1-OST0005_UUID@10.214.127.185@tcp:28/4 lens 368/584 e 0 to 1 dl 1339994155 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 18 06:35:55 sklusp01a kernel: Lustre: 11293:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jun 18 06:36:02 sklusp01a kernel: LustreError: 11-0: an error occurred while communicating with 10.214.127.57@tcp. The ost_connect operation failed with -19 Jun 18 06:36:02 sklusp01a kernel: LustreError: Skipped 1 previous similar message Jun 18 06:36:12 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) l1-OST0005-osc: tried all connections, increasing latency to 5s Jun 18 06:36:12 sklusp01a kernel: Lustre: 11294:0:(import.c:517:import_select_connection()) Skipped 3 previous similar messages Jun 18 06:36:12 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Only 0/6 OSTs are active, abort quota recovery Jun 18 06:36:12 sklusp01a kernel: Lustre: 11293:0:(quota_master.c:1716:mds_quota_recovery()) Skipped 2 previous similar messages Jun 18 06:36:12 sklusp01a kernel: Lustre: MDS l1-MDT0000: l1-OST0005_UUID now active, resetting orphans Jun 18 06:36:12 sklusp01a kernel: Lustre: Skipped 2 previous similar messages Jun 18 06:36:12 sklusp01a kernel: Lustre: l1-OST0005-osc: Connection restored to service l1-OST0005 using nid 10.214.127.185@tcp. Jun 18 06:36:12 sklusp01a kernel: Lustre: Skipped 2 previous similar messages Jun 18 15:30:54 sklusp01a kernel: Lustre: Host 10.214.127.215 reset our connection while we were sending data; it may have rebooted.