Oct 30 09:51:17 barry-mds1 kernel: [92265.082335] Lustre: lustre-MDT0000: haven't heard from client 70001521-339c-6de9-81d6-d8a4cbeb8701 (at 30@gni1) in 160 seconds. I think it's dead, and I am evicting it. exp ffff880290005400, cur 1351605077 expire 1351604975 last 1351604917
Oct 30 09:51:17 barry-mds1 kernel: [92265.104054] Lustre: lustre-MDT0000: haven't heard from client 8bd73557-b848-4b66-97ae-5423b0c609b7 (at 26@gni1) in 160 seconds. I think it's dead, and I am evicting it. exp ffff8803e6e2f000, cur 1351605077 expire 1351604975 last 1351604917
Oct 30 09:57:18 barry-mds1 kernel: [92626.056019] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 21 seconds
Oct 30 09:57:18 barry-mds1 kernel: [92626.057001] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Timed out RDMA with 10.37.202.60@o2ib1 (50): c: 0, oc: 0, rc: 8
Oct 30 09:57:18 barry-mds1 kernel: [92626.099265] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds
Oct 30 09:57:18 barry-mds1 kernel: [92626.100255] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Timed out RDMA with 10.37.202.61@o2ib1 (50): c: 0, oc: 0, rc: 8
Oct 30 09:58:08 barry-mds1 kernel: [92676.056025] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 18 seconds
Oct 30 09:58:08 barry-mds1 kernel: [92676.057001] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Timed out RDMA with 10.37.202.59@o2ib1 (50): c: 0, oc: 0, rc: 8
Oct 30 11:05:12 barry-mds1 kernel: [96700.372643] Lustre: lustre-MDT0000: haven't heard from client d9098acb-e619-9fee-5e0c-1452f3419b13 (at 26@gni1) in 163 seconds. I think it's dead, and I am evicting it. exp ffff880290ba5800, cur 1351609512 expire 1351609410 last 1351609349
Oct 30 11:10:57 barry-mds1 kernel: [97045.056058] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds
Oct 30 11:10:57 barry-mds1 kernel: [97045.057042] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Timed out RDMA with 10.37.202.59@o2ib1 (51): c: 0, oc: 0, rc: 8
Oct 30 11:11:47 barry-mds1 kernel: [97095.056061] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 9 seconds
Oct 30 11:11:47 barry-mds1 kernel: [97095.057042] LNetError: 8578:0:(o2iblnd_cb.c:3009:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Oct 30 11:11:47 barry-mds1 kernel: [97095.057042] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Timed out RDMA with 10.37.202.60@o2ib1 (49): c: 0, oc: 0, rc: 8
Oct 30 11:11:47 barry-mds1 kernel: [97095.057042] LNetError: 8578:0:(o2iblnd_cb.c:3072:kiblnd_check_conns()) Skipped 1 previous similar message
Oct 30 13:40:38 barry-mds1 kernel: [106025.549013] LustreError: 9728:0:(osp_dev.c:337:osp_statfs()) ASSERTION( sfs->os_fprecreated <= 20000 ) failed: 
Oct 30 13:40:38 barry-mds1 kernel: [106025.549018] LustreError: 9723:0:(osp_dev.c:337:osp_statfs()) ASSERTION( sfs->os_fprecreated <= 20000 ) failed: 
Oct 30 13:40:38 barry-mds1 kernel: [106025.549021] LustreError: 9723:0:(osp_dev.c:337:osp_statfs()) LBUG
Oct 30 13:40:38 barry-mds1 kernel: [106025.549023] Pid: 9723, comm: mdt01_490
Oct 30 13:40:38 barry-mds1 kernel: [106025.549024] 
Oct 30 13:40:38 barry-mds1 kernel: [106025.549024] Call Trace:
Oct 30 13:40:38 barry-mds1 kernel: [106025.549069]  [<ffffffffa04fc905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549082]  [<ffffffffa04fcf17>] lbug_with_loc+0x47/0xb0 [libcfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549092]  [<ffffffffa09fa7da>] osp_statfs+0x28a/0x2a0 [osp]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549112]  [<ffffffffa0815e8e>] lod_statfs_and_check+0x8e/0x4a0 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549121]  [<ffffffffa0818430>] lod_alloc_rr.clone.2+0x500/0xd90 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549130]  [<ffffffffa081a04c>] lod_qos_prep_create+0xe5c/0x1848 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549162]  [<ffffffffa0e5e692>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549171]  [<ffffffffa0814c7b>] lod_declare_striped_object+0x14b/0x920 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549180]  [<ffffffffa0815758>] lod_declare_object_create+0x308/0x4f0 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549215]  [<ffffffffa0dd9ccf>] mdd_declare_object_create_internal+0xaf/0x1d0 [mdd]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549231] LustreError: 9225:0:(osp_dev.c:337:osp_statfs()) ASSERTION( sfs->os_fprecreated <= 20000 ) failed: 
Oct 30 13:40:38 barry-mds1 kernel: [106025.549235]  [<ffffffffa0deac9a>] mdd_create+0x39a/0x1550 [mdd]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549238] LustreError: 9225:0:(osp_dev.c:337:osp_statfs()) LBUG
Oct 30 13:40:38 barry-mds1 kernel: [106025.549240] Pid: 9225, comm: mdt00_504
Oct 30 13:40:38 barry-mds1 kernel: [106025.549241] 
Oct 30 13:40:38 barry-mds1 kernel: [106025.549241] Call Trace:
Oct 30 13:40:38 barry-mds1 kernel: [106025.549276]  [<ffffffffa0eee519>] mdt_reint_open+0x1079/0x1860 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549286]  [<ffffffffa04fc905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549288]  [<ffffffffa0df0bee>] ? md_ucred+0x1e/0x60 [mdd]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549300]  [<ffffffffa04fcf17>] lbug_with_loc+0x47/0xb0 [libcfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549302]  [<ffffffffa0eb9765>] ? mdt_ucred+0x15/0x20 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549312]  [<ffffffffa09fa7da>] osp_statfs+0x28a/0x2a0 [osp]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549316]  [<ffffffffa0ed93e1>] mdt_reint_rec+0x41/0xe0 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549329]  [<ffffffffa0ed2b73>] mdt_reint_internal+0x4f3/0x7f0 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549339]  [<ffffffffa0815e8e>] lod_statfs_and_check+0x8e/0x4a0 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549343]  [<ffffffffa0ed313d>] mdt_intent_reint+0x1ed/0x500 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549356]  [<ffffffffa0818430>] lod_alloc_rr.clone.2+0x500/0xd90 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549359]  [<ffffffffa0eceb0e>] mdt_intent_policy+0x38e/0x770 [mdt]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549367]  [<ffffffffa081a04c>] lod_qos_prep_create+0xe5c/0x1848 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549413]  [<ffffffffa0e5e692>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549422]  [<ffffffffa0814c7b>] lod_declare_striped_object+0x14b/0x920 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549431]  [<ffffffffa0815758>] lod_declare_object_create+0x308/0x4f0 [lod]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549474]  [<ffffffffa0dd9ccf>] mdd_declare_object_create_internal+0xaf/0x1d0 [mdd]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549477]  [<ffffffffa0861b21>] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc]
Oct 30 13:40:38 barry-mds1 kernel: [106025.549486]  [<ffffffffa0deac9a>] mdd_create+0x39a/0x1550 [mdd]