Mar 17 08:42:51 r3169 kernel: imklog 5.8.10, log source = /proc/kmsg started. Mar 17 08:42:51 r3169 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3151" x-info="http://www.rsyslog.com"] start Mar 17 08:42:51 r3169 kernel: kjournald starting. Commit interval 5 seconds Mar 17 08:42:51 r3169 kernel: EXT3-fs (sda1): mounted filesystem with ordered data mode Mar 17 08:42:51 r3169 kernel: readahead-collector: starting Mar 17 08:42:51 r3169 kernel: udev: starting version 147 Mar 17 08:42:51 r3169 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18 Mar 17 08:42:51 r3169 kernel: i801_smbus 0000:40:00.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18 Mar 17 08:42:51 r3169 kernel: i801_smbus 0000:40:00.3: Enabling SMBus device Mar 17 08:42:51 r3169 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Mar 17 08:42:51 r3169 kernel: iTCO_vendor_support: vendor-support=0 Mar 17 08:42:51 r3169 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh Mar 17 08:42:51 r3169 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS Mar 17 08:42:51 r3169 kernel: EDAC MC: Ver: 2.1.0 Dec 19 2012 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Mar 17 08:42:51 r3169 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0 Mar 17 08:42:51 r3169 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0 Mar 17 08:42:51 r3169 kernel: EDAC sbridge: Driver loaded. Mar 17 08:42:51 r3169 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU12 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU13 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU14 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: microcode: CPU15 sig=0x206d7, pf=0x1, revision=0x70c Mar 17 08:42:51 r3169 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07 Mar 17 08:42:51 r3169 kernel: Microcode Update Driver: v2.00 , Peter Oruba Mar 17 08:42:51 r3169 kernel: Built 2 zonelists in Node order, mobility grouping on. Total pages: 16500652 Mar 17 08:42:51 r3169 kernel: Policy zone: Normal Mar 17 08:42:51 r3169 kernel: EXT3-fs (sda1): using internal journal Mar 17 08:42:51 r3169 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team Mar 17 08:42:51 r3169 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max) Mar 17 08:42:51 r3169 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Mar 17 08:42:51 r3169 kernel: ib0: enabling connected mode will cause multicast packet drops Mar 17 08:42:51 r3169 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready Mar 17 08:42:51 r3169 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Mar 17 08:42:51 r3169 rpc.statd[3210]: Version 1.2.3 starting Mar 17 08:42:51 r3169 sm-notify[3211]: Version 1.2.3 starting Mar 17 08:42:52 r3169 kernel: Lustre: Lustre: Build Version: RC2--PRISTINE-2.6.32-279.19.1.el6.x86_64 Mar 17 08:42:52 r3169 kernel: Lustre: Added LNI 10.9.55.1@o2ib3 [8/64/0/180] Mar 17 08:42:53 r3169 kernel: Lustre: Lustre OSC module (ffffffffa0e9b880). Mar 17 08:42:53 r3169 kernel: Lustre: Lustre LOV module (ffffffffa0f2dce0). Mar 17 08:42:53 r3169 kernel: Lustre: Lustre client module (ffffffffa1019020). Mar 17 08:42:53 r3169 kernel: Lustre: MGC10.9.103.1@o2ib3: Reactivating import Mar 17 08:42:53 r3169 kernel: LustreError: 929:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:42:53 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470173/real 1363470173] req@ffff88062749b400 x1429702099075137/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470178 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:42:53 r3169 kernel: Lustre: Mounted images-client Mar 17 08:42:53 r3169 acpid: starting up Mar 17 08:42:53 r3169 acpid: 1 rule loaded Mar 17 08:42:53 r3169 acpid: waiting for events: event logging is off Mar 17 08:42:53 r3169 mcelog: failed to prefill DIMM database from DMI data Mar 17 08:42:54 r3169 xinetd[3393]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. Mar 17 08:42:54 r3169 xinetd[3393]: Started working: 2 available services Mar 17 08:42:54 r3169 ntpdate[3404]: step time server 10.10.105.1 offset 0.000969 sec Mar 17 08:42:54 r3169 ntpd[3411]: ntpd 4.2.4p8@1.1612-o Tue Nov 29 00:09:12 UTC 2011 (1) Mar 17 08:42:54 r3169 ntpd[3412]: precision = 0.066 usec Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #1 wildcard, ::#123 Disabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #2 ib0, fe80::202:c903:17:1ee1#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #3 lo, ::1#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #4 eth0, fe80::67d:7bff:fefd:36bb#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #5 lo, 127.0.0.1#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #6 eth0, 10.10.55.1#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on interface #7 ib0, 10.9.55.1#123 Enabled Mar 17 08:42:54 r3169 ntpd[3412]: Listening on routing socket on fd #24 for interface updates Mar 17 08:42:54 r3169 ntpd[3412]: kernel time sync status 2040 Mar 17 08:42:57 r3169 CMDaemon: Info: CMDaemon version 1.4 (r16277) Mar 17 08:42:57 r3169 CMDaemon: Info: CM version 6.0 Mar 17 08:42:57 r3169 CMDaemon: Info: Reading configuration from /cm/local/apps/cmd/etc/cmd.conf Mar 17 08:43:09 r3169 kernel: type=1305 audit(1363470189.076:32268): auid=4294967295 ses=4294967295 subj=kernel op="remove rule" key=(null) list=4 res=1 Mar 17 08:43:09 r3169 kernel: type=1305 audit(1363470189.076:32269): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=kernel res=1 Mar 17 08:43:09 r3169 kernel: readahead-collector: sorting Mar 17 08:43:09 r3169 kernel: readahead-collector: finished Mar 17 08:43:43 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:44:08 r3169 kernel: LustreError: 927:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:44:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470248/real 1363470248] req@ffff881064484800 x1429702099075267/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470259 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:44:33 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:44:58 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:44:58 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470298/real 1363470298] req@ffff88100a3e6800 x1429702099078280/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470314 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:45:23 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:45:48 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:45:48 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470348/real 1363470348] req@ffff88100a019000 x1429702099078408/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470369 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:46:13 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:46:38 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:46:38 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470398/real 1363470398] req@ffff881069962c00 x1429702099078512/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:47:03 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:47:28 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:47:28 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470448/real 1363470448] req@ffff88106cfe6800 x1429702099078616/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470479 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:47:53 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:48:18 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:48:18 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470498/real 1363470498] req@ffff881063e8c000 x1429702099078720/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470534 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:48:43 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:49:08 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:49:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470548/real 1363470548] req@ffff88100a072c00 x1429702099078824/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470589 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:49:33 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:49:38 r3169 kernel: LustreError: 3714:0:(namei.c:1128:ll_objects_destroy()) obd destroy objid 0xef error -5 Mar 17 08:49:38 r3169 kernel: LustreError: 3714:0:(file.c:166:ll_close_inode_openhandle()) inode 144120422969901295 ll_objects destroy: rc = -5 Mar 17 08:49:58 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:49:58 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470598/real 1363470598] req@ffff881069f1c000 x1429702099080752/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470644 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:50:23 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:51:38 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:51:38 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) Skipped 1 previous similar message Mar 17 08:51:38 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470698/real 1363470698] req@ffff881049503000 x1429702099081009/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:51:38 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 17 08:52:03 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:52:03 r3169 kernel: LustreError: Skipped 1 previous similar message Mar 17 08:53:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 08:53:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:53:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 08:53:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 08:53:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 08:53:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 08:53:26 r3169 kernel: Call Trace: Mar 17 08:53:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:53:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:53:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:53:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:53:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:53:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:53:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:53:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:53:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:53:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:53:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:53:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:53:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:53:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:53:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:53:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:53:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:53:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:53:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:53:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:53:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:53:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:53:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:53:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:53:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:53:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:53:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:53:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:53:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:53:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:53:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:53:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:54:08 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:54:08 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) Skipped 2 previous similar messages Mar 17 08:54:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470848/real 1363470848] req@ffff881063ce2000 x1429702099081321/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470904 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:54:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 17 08:54:33 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:54:33 r3169 kernel: LustreError: Skipped 2 previous similar messages Mar 17 08:54:39 r3169 kernel: LustreError: 3777:0:(namei.c:1128:ll_objects_destroy()) obd destroy objid 0x126 error -5 Mar 17 08:54:39 r3169 kernel: LustreError: 3777:0:(file.c:166:ll_close_inode_openhandle()) inode 144120422969901350 ll_objects destroy: rc = -5 Mar 17 08:55:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 08:55:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:55:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 08:55:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 08:55:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 08:55:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 08:55:26 r3169 kernel: Call Trace: Mar 17 08:55:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:55:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:55:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:55:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:55:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:55:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:55:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:55:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:55:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:55:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:55:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:55:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:55:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:55:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:55:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:55:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:55:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:55:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:55:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:55:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:55:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:55:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:55:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:55:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:55:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:55:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:55:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:55:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:55:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:55:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:55:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:55:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:57:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 08:57:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:57:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 08:57:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 08:57:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 08:57:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 08:57:26 r3169 kernel: Call Trace: Mar 17 08:57:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:57:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:57:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:57:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:57:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:57:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:57:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:57:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:57:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:57:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:57:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:57:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:57:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:57:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:57:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:57:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:57:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:57:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:57:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:57:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:57:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:57:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:57:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:57:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:57:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:57:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:57:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:57:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:57:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:57:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:57:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:57:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:57:26 r3169 kernel: INFO: task rsync:3777 blocked for more than 120 seconds. Mar 17 08:57:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:57:26 r3169 kernel: rsync D 000000000000000c 0 3777 1 0x00000080 Mar 17 08:57:26 r3169 kernel: ffff881004b53988 0000000000000046 0000000000000000 ffff881012389560 Mar 17 08:57:26 r3169 kernel: ffff881069dcbb58 ffff88086bfda680 ffff8810123895f8 ffff881004b53948 Mar 17 08:57:26 r3169 kernel: ffff881063ecbaf8 ffff881004b53fd8 000000000000fb88 ffff881063ecbaf8 Mar 17 08:57:26 r3169 kernel: Call Trace: Mar 17 08:57:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:57:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:57:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:57:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:57:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:57:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:57:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:57:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:57:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:57:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:57:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:57:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:57:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:57:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:57:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:57:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:57:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:57:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:57:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:57:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:57:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:57:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:57:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:57:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:57:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:57:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:57:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:57:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:57:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:57:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:57:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:57:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:59:08 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:59:08 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) Skipped 5 previous similar messages Mar 17 08:59:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363471148/real 1363471148] req@ffff88106c34f000 x1429702099083249/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363471204 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:59:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 17 08:59:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 08:59:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:59:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 08:59:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 08:59:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 08:59:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 08:59:26 r3169 kernel: Call Trace: Mar 17 08:59:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:59:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:59:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:59:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:59:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:59:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:59:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:59:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:59:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:59:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:59:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:59:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:59:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:59:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:59:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:59:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:59:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:59:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:59:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:59:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:59:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:59:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:59:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:59:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:59:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:59:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:59:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:59:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:59:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:59:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:59:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:59:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:59:26 r3169 kernel: INFO: task rsync:3777 blocked for more than 120 seconds. Mar 17 08:59:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 08:59:26 r3169 kernel: rsync D 000000000000000c 0 3777 1 0x00000080 Mar 17 08:59:26 r3169 kernel: ffff881004b53988 0000000000000046 0000000000000000 ffff881012389560 Mar 17 08:59:26 r3169 kernel: ffff881069dcbb58 ffff88086bfda680 ffff8810123895f8 ffff881004b53948 Mar 17 08:59:26 r3169 kernel: ffff881063ecbaf8 ffff881004b53fd8 000000000000fb88 ffff881063ecbaf8 Mar 17 08:59:26 r3169 kernel: Call Trace: Mar 17 08:59:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 08:59:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 08:59:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 08:59:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 08:59:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 08:59:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 08:59:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 08:59:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 08:59:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 08:59:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 08:59:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 08:59:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 08:59:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 08:59:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 08:59:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 08:59:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 08:59:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 08:59:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 08:59:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 08:59:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 08:59:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 08:59:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 08:59:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 08:59:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 08:59:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 08:59:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 08:59:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 08:59:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 08:59:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 08:59:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 08:59:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 08:59:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 08:59:33 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:59:33 r3169 kernel: LustreError: Skipped 5 previous similar messages Mar 17 08:59:44 r3169 kernel: LustreError: 3818:0:(namei.c:1128:ll_objects_destroy()) obd destroy objid 0x164 error -5 Mar 17 08:59:44 r3169 kernel: LustreError: 3818:0:(file.c:166:ll_close_inode_openhandle()) inode 144120422969901412 ll_objects destroy: rc = -5 Mar 17 09:01:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 09:01:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 09:01:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 09:01:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 09:01:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 09:01:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 09:01:26 r3169 kernel: Call Trace: Mar 17 09:01:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 09:01:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 09:01:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 09:01:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 09:01:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 09:01:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 09:01:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 09:01:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 09:01:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 09:01:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 09:01:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 09:01:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 09:01:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 09:01:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 09:01:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 09:01:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 09:01:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 09:01:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 09:01:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 09:01:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 09:01:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 09:01:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 09:01:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 09:01:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 09:01:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 09:01:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 09:01:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 09:01:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 09:01:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 09:01:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 09:01:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 09:01:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 09:01:26 r3169 kernel: INFO: task rsync:3777 blocked for more than 120 seconds. Mar 17 09:01:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 09:01:26 r3169 kernel: rsync D 000000000000000c 0 3777 1 0x00000080 Mar 17 09:01:26 r3169 kernel: ffff881004b53988 0000000000000046 0000000000000000 ffff881012389560 Mar 17 09:01:26 r3169 kernel: ffff881069dcbb58 ffff88086bfda680 ffff8810123895f8 ffff881004b53948 Mar 17 09:01:26 r3169 kernel: ffff881063ecbaf8 ffff881004b53fd8 000000000000fb88 ffff881063ecbaf8 Mar 17 09:01:26 r3169 kernel: Call Trace: Mar 17 09:01:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 09:01:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 09:01:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 09:01:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 09:01:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 09:01:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 09:01:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 09:01:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 09:01:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 09:01:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 09:01:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 09:01:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 09:01:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 09:01:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 09:01:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 09:01:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 09:01:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 09:01:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 09:01:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 09:01:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 09:01:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 09:01:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 09:01:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 09:01:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 09:01:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 09:01:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 09:01:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 09:01:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 09:01:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 09:01:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 09:01:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 09:01:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 09:03:26 r3169 kernel: INFO: task rsync:3714 blocked for more than 120 seconds. Mar 17 09:03:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 09:03:26 r3169 kernel: rsync D 000000000000000b 0 3714 1 0x00000080 Mar 17 09:03:26 r3169 kernel: ffff88100a131988 0000000000000046 0000000000000000 ffff8810651be700 Mar 17 09:03:26 r3169 kernel: ffff8808697c0b58 ffff88086bfda680 ffff8810651be798 ffff88100a131948 Mar 17 09:03:26 r3169 kernel: ffff881069b485f8 ffff88100a131fd8 000000000000fb88 ffff881069b485f8 Mar 17 09:03:26 r3169 kernel: Call Trace: Mar 17 09:03:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 09:03:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 09:03:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 09:03:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 09:03:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 09:03:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 09:03:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 09:03:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 09:03:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 09:03:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 09:03:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 09:03:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 09:03:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 09:03:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 09:03:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 09:03:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 09:03:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 09:03:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 09:03:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 09:03:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 09:03:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 09:03:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 09:03:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 09:03:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 09:03:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 09:03:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 09:03:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 09:03:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 09:03:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 09:03:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 09:03:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 09:03:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 09:03:26 r3169 kernel: INFO: task rsync:3777 blocked for more than 120 seconds. Mar 17 09:03:26 r3169 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 17 09:03:26 r3169 kernel: rsync D 000000000000000c 0 3777 1 0x00000080 Mar 17 09:03:26 r3169 kernel: ffff881004b53988 0000000000000046 0000000000000000 ffff881012389560 Mar 17 09:03:26 r3169 kernel: ffff881069dcbb58 ffff88086bfda680 ffff8810123895f8 ffff881004b53948 Mar 17 09:03:26 r3169 kernel: ffff881063ecbaf8 ffff881004b53fd8 000000000000fb88 ffff881063ecbaf8 Mar 17 09:03:26 r3169 kernel: Call Trace: Mar 17 09:03:26 r3169 kernel: [] cfs_waitq_wait+0xe/0x10 [libcfs] Mar 17 09:03:26 r3169 kernel: [] lov_delete_raid0+0x2e4/0x400 [lov] Mar 17 09:03:26 r3169 kernel: [] lov_object_delete+0x66/0x190 [lov] Mar 17 09:03:26 r3169 kernel: [] lu_object_free+0x89/0x1b0 [obdclass] Mar 17 09:03:26 r3169 kernel: [] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Mar 17 09:03:26 r3169 kernel: [] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs] Mar 17 09:03:26 r3169 kernel: [] lu_object_put+0x1bf/0x290 [obdclass] Mar 17 09:03:26 r3169 kernel: [] ? cl_inode_fini+0x52/0x270 [lustre] Mar 17 09:03:26 r3169 kernel: [] cl_object_put+0xe/0x10 [obdclass] Mar 17 09:03:26 r3169 kernel: [] cl_inode_fini+0x8a/0x270 [lustre] Mar 17 09:03:26 r3169 kernel: [] ? lmv_change_cbdata+0x2cc/0x580 [lmv] Mar 17 09:03:26 r3169 kernel: [] ? ldlm_resource_iterate+0x82/0x1e0 [ptlrpc] Mar 17 09:03:26 r3169 kernel: [] ll_clear_inode+0x2ff/0xdb0 [lustre] Mar 17 09:03:26 r3169 kernel: [] ? ll_delete_inode+0x0/0x1a0 [lustre] Mar 17 09:03:26 r3169 kernel: [] clear_inode+0xac/0x140 Mar 17 09:03:26 r3169 kernel: [] ll_delete_inode+0x76/0x1a0 [lustre] Mar 17 09:03:26 r3169 kernel: [] generic_delete_inode+0xde/0x1d0 Mar 17 09:03:26 r3169 kernel: [] generic_drop_inode+0x65/0x80 Mar 17 09:03:26 r3169 kernel: [] iput+0x62/0x70 Mar 17 09:03:26 r3169 kernel: [] ll_d_iput+0x31/0x80 [lustre] Mar 17 09:03:26 r3169 kernel: [] dentry_iput+0x7c/0x100 Mar 17 09:03:26 r3169 kernel: [] d_kill+0x31/0x60 Mar 17 09:03:26 r3169 kernel: [] dput+0x7c/0x150 Mar 17 09:03:26 r3169 kernel: [] __fput+0x189/0x210 Mar 17 09:03:26 r3169 kernel: [] fput+0x25/0x30 Mar 17 09:03:26 r3169 kernel: [] filp_close+0x5d/0x90 Mar 17 09:03:26 r3169 kernel: [] put_files_struct+0x7f/0xf0 Mar 17 09:03:26 r3169 kernel: [] exit_files+0x53/0x70 Mar 17 09:03:26 r3169 kernel: [] do_exit+0x18d/0x870 Mar 17 09:03:26 r3169 kernel: [] do_group_exit+0x58/0xd0 Mar 17 09:03:26 r3169 kernel: [] sys_exit_group+0x17/0x20 Mar 17 09:03:26 r3169 kernel: [] system_call_fastpath+0x16/0x1b Mar 17 09:04:47 r3169 kernel: LustreError: 3882:0:(namei.c:1128:ll_objects_destroy()) obd destroy objid 0x1bd error -5 Mar 17 09:04:47 r3169 kernel: LustreError: 3882:0:(file.c:166:ll_close_inode_openhandle()) inode 144120422969901501 ll_objects destroy: rc = -5 Mar 17 09:08:18 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 09:08:18 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) Skipped 10 previous similar messages Mar 17 09:08:18 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363471698/real 1363471698] req@ffff88100a329000 x1429702099087531/t0(0) o8->images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363471749 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 09:08:18 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 17 09:08:43 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 09:08:43 r3169 kernel: LustreError: Skipped 10 previous similar messages Mar 17 09:09:51 r3169 kernel: LustreError: 3972:0:(namei.c:1128:ll_objects_destroy()) obd destroy objid 0x201 error -5 Mar 17 09:09:51 r3169 kernel: LustreError: 3972:0:(file.c:166:ll_close_inode_openhandle()) inode 144120422969901569 ll_objects destroy: rc = -5