Mar 9 00:22:31 widow-oss11c2 kernel: [41677.119862] LustreError: 16402:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 126385280 != fo_tot_granted 128482432 Mar 9 00:22:31 widow-oss11c2 kernel: [41677.158873] LustreError: 16302:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 151551104 != fo_tot_granted 153648256 Mar 9 00:22:33 widow-oss11c2 kernel: [41679.262487] LustreError: 16510:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 67665024 != fo_tot_granted 71859328 Mar 9 00:36:21 widow-oss11c2 kernel: [42507.517044] Lustre: 25533:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362807101/real 1362807101] req@ffff880426d51800 x1429003376266444/t0(0) o39->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362807381 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 00:36:21 widow-oss11c2 kernel: [42507.596054] LustreError: 25533:0:(obd_mount.c:2115:server_put_super()) routed1-OST0051: failed to disconnect lwp. (rc=-110) Mar 9 00:36:21 widow-oss11c2 kernel: [42507.625209] Lustre: Failing over routed1-OST0051 Mar 9 00:36:22 widow-oss11c2 kernel: [42508.168000] Lustre: server umount routed1-OST0051 complete Mar 9 00:36:22 widow-oss11c2 kernel: [42508.208351] Lustre: 25582:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362807102/real 1362807102] req@ffff880260632800 x1429003376266448/t0(0) o39->routed1-MDT0000-lwp-OST00b1@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362807382 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 00:36:22 widow-oss11c2 kernel: [42508.287201] LustreError: 25582:0:(obd_mount.c:2115:server_put_super()) routed1-OST00b1: failed to disconnect lwp. (rc=-110) Mar 9 00:36:22 widow-oss11c2 kernel: [42508.316329] Lustre: Failing over routed1-OST00b1 Mar 9 00:36:23 widow-oss11c2 kernel: [42508.790254] Lustre: server umount routed1-OST00b1 complete Mar 9 00:36:23 widow-oss11c2 kernel: [42508.880042] Lustre: 25637:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362807103/real 1362807103] req@ffff88034024fc00 x1429003376266452/t0(0) o39->routed1-MDT0000-lwp-OST0111@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362807383 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 00:36:23 widow-oss11c2 kernel: [42508.959083] LustreError: 25637:0:(obd_mount.c:2115:server_put_super()) routed1-OST0111: failed to disconnect lwp. (rc=-110) Mar 9 00:36:23 widow-oss11c2 kernel: [42508.988488] Lustre: Failing over routed1-OST0111 Mar 9 00:36:23 widow-oss11c2 kernel: [42509.441118] Lustre: server umount routed1-OST0111 complete Mar 9 00:36:24 widow-oss11c2 kernel: [42509.939041] Lustre: 25743:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362807104/real 1362807104] req@ffff88026adea800 x1429003376266460/t0(0) o39->routed1-MDT0000-lwp-OST01d1@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362807384 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 00:36:24 widow-oss11c2 kernel: [42510.019355] Lustre: 25743:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 9 00:36:24 widow-oss11c2 kernel: [42510.047127] LustreError: 25743:0:(obd_mount.c:2115:server_put_super()) routed1-OST01d1: failed to disconnect lwp. (rc=-110) Mar 9 00:36:24 widow-oss11c2 kernel: [42510.077250] LustreError: 25743:0:(obd_mount.c:2115:server_put_super()) Skipped 1 previous similar message Mar 9 00:36:24 widow-oss11c2 kernel: [42510.104341] Lustre: Failing over routed1-OST01d1 Mar 9 00:36:24 widow-oss11c2 kernel: [42510.121456] Lustre: Skipped 1 previous similar message Mar 9 00:36:24 widow-oss11c2 kernel: [42510.568562] Lustre: server umount routed1-OST01d1 complete Mar 9 00:36:24 widow-oss11c2 kernel: [42510.587409] Lustre: Skipped 1 previous similar message Mar 9 00:41:04 widow-oss11c2 kernel: [42790.669030] Lustre: 25800:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362807384/real 1362807384] req@ffff8804267fdc00 x1429003376266480/t0(0) o251->MGC10.36.227.201@o2ib@10.36.227.201@o2ib:26/25 lens 224/224 e 0 to 1 dl 1362807664 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 00:41:04 widow-oss11c2 kernel: [42790.748959] Lustre: 25800:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 9 00:41:05 widow-oss11c2 kernel: [42791.247348] Lustre: server umount routed1-OST0231 complete Mar 9 00:41:30 widow-oss11c2 kernel: [42816.645071] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:41:31 widow-oss11c2 kernel: [42817.670063] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:41:33 widow-oss11c2 kernel: [42819.695069] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:41:37 widow-oss11c2 kernel: [42823.720067] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:41:45 widow-oss11c2 kernel: [42831.745067] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:42:01 widow-oss11c2 kernel: [42847.770067] LNet: 10338:0:(router.c:1215:lnet_prune_rc_data()) Waiting for rc buffers to unlink Mar 9 00:42:06 widow-oss11c2 kernel: [42852.545102] LNet: 2708:0:(o2iblnd.c:2912:kiblnd_shutdown()) 10.36.227.130@o2ib: waiting for 1 peers to disconnect Mar 9 00:42:10 widow-oss11c2 kernel: [42856.573041] LNet: 2708:0:(o2iblnd.c:2912:kiblnd_shutdown()) 10.36.227.130@o2ib: waiting for 1 peers to disconnect Mar 9 00:42:18 widow-oss11c2 kernel: [42864.601045] LNet: 2708:0:(o2iblnd.c:2912:kiblnd_shutdown()) 10.36.227.130@o2ib: waiting for 1 peers to disconnect Mar 9 00:42:34 widow-oss11c2 kernel: [42880.629068] LNet: 2708:0:(o2iblnd.c:2912:kiblnd_shutdown()) 10.36.227.130@o2ib: waiting for 1 peers to disconnect Mar 9 00:43:06 widow-oss11c2 kernel: [42912.657053] LNet: 2708:0:(o2iblnd.c:2912:kiblnd_shutdown()) 10.36.227.130@o2ib: waiting for 1 peers to disconnect Mar 9 00:43:43 widow-oss11c2 kernel: [42949.713041] LNet: Removed LNI 10.36.227.130@o2ib Mar 9 00:52:11 widow-oss11c2 kernel: [43457.355040] LNet: HW CPU cores: 8, npartitions: 2 Mar 9 00:52:11 widow-oss11c2 kernel: [43457.371781] alg: No test for crc32 (crc32-table) Mar 9 00:52:11 widow-oss11c2 kernel: [43457.388116] alg: No test for adler32 (adler32-zlib) Mar 9 00:52:19 widow-oss11c2 kernel: [43465.532832] Lustre: Lustre: Build Version: 2.3.62-g0be1c87-CHANGED-2.6.32-279.22.1.el6.head.x86_64 Mar 9 00:52:20 widow-oss11c2 kernel: [43466.401982] LNet: Added LNI 10.36.227.130@o2ib [8/2560/0/180] Mar 9 00:53:27 widow-oss11c2 kernel: [43533.494514] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:53:28 widow-oss11c2 kernel: [43534.133613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:53:30 widow-oss11c2 kernel: [43536.154803] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:53:30 widow-oss11c2 kernel: [43536.773614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:53:31 widow-oss11c2 kernel: [43537.037949] LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:53:31 widow-oss11c2 kernel: [43537.252674] LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. quota=on. Opts: Mar 9 00:58:26 widow-oss11c2 kernel: [43832.539606] Lustre: routed1-OST0051: deleting orphan objects from 0x0:25914 to 26016 Mar 9 00:58:26 widow-oss11c2 kernel: [43832.563105] Lustre: routed1-OST00b1: deleting orphan objects from 0x0:25934 to 26016 Mar 9 00:58:26 widow-oss11c2 kernel: [43832.584496] Lustre: Skipped 1 previous similar message Mar 9 00:58:26 widow-oss11c2 kernel: [43832.689749] LustreError: 6620:0:(ldlm_resource.c:1161:ldlm_resource_get()) lvbo_init failed for resource 25984: rc -2 Mar 9 00:58:26 widow-oss11c2 kernel: [43832.716832] LustreError: 6620:0:(ldlm_resource.c:1161:ldlm_resource_get()) lvbo_init failed for resource 25983: rc -2 Mar 9 03:32:38 widow-oss11c2 kernel: [53084.704023] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 15265@gni ns: filter-ffff8802e8b73000 lock: ffff8803a10e7a00/0x96f970681903b770 lrc: 3/0,0 mode: PW/PW res: 28520/0 rrc: 3 type: EXT [196608->262143] (req 196608->262143) flags: 0x20 nid: 15265@gni remote: 0xc5e1b8db1183f4dd expref: 4 pid: 9962 timeout: 4347751586 lvb_type: 0 Mar 9 03:32:38 widow-oss11c2 kernel: [53084.798738] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 15379@gni ns: filter-ffff8802e8b73000 lock: ffff880378b8ee00/0x96f970681903b769 lrc: 3/0,0 mode: PW/PW res: 28520/0 rrc: 3 type: EXT [131072->196607] (req 131072->196607) flags: 0x20 nid: 15379@gni remote: 0xa5c43920a3ce5c40 expref: 4 pid: 9255 timeout: 4347751586 lvb_type: 0 Mar 9 04:04:39 widow-oss11c2 kernel: [55005.256791] Lustre: routed1-OST0051: Client c6460c8b-bc6a-e747-ddba-b89e3ce3264c (at 15817@gni) reconnecting Mar 9 04:04:58 widow-oss11c2 kernel: [55024.682195] Lustre: routed1-OST01d1: Client c3a02403-559d-8768-5406-5a88b439b6f3 (at 16500@gni) reconnecting Mar 9 04:05:00 widow-oss11c2 kernel: [55025.807411] Lustre: routed1-OST0111: Client 97193b04-5964-c2dd-758f-32ee75ac69ed (at 16445@gni) reconnecting Mar 9 04:05:02 widow-oss11c2 kernel: [55028.754405] Lustre: routed1-OST0051: Client 893bbb17-c5d2-30cf-7643-f908b82700c9 (at 16447@gni) reconnecting Mar 9 04:05:23 widow-oss11c2 kernel: [55049.377385] Lustre: routed1-OST0231: Client ae05f929-5eab-5633-650f-5634c8f0e911 (at 15052@gni) reconnecting Mar 9 04:05:23 widow-oss11c2 kernel: [55049.402768] Lustre: Skipped 2 previous similar messages Mar 9 04:06:09 widow-oss11c2 kernel: [55095.027313] Lustre: routed1-OST0051: Client fcc2ae5f-34ef-e10e-43b1-1fc26e66e54f (at 16499@gni) reconnecting Mar 9 04:06:09 widow-oss11c2 kernel: [55095.053077] Lustre: Skipped 2 previous similar messages Mar 9 04:06:32 widow-oss11c2 kernel: [55118.574705] Lustre: routed1-OST01d1: Client 82b8a686-a586-75af-c58b-79815ae1f4b7 (at 463@gni) reconnecting Mar 9 04:06:32 widow-oss11c2 kernel: [55118.599862] Lustre: Skipped 1 previous similar message Mar 9 04:06:55 widow-oss11c2 kernel: [55141.270551] Lustre: routed1-OST0231: Client d3b2462f-84e3-3abe-4a22-35aef3ff85df (at 15945@gni) reconnecting Mar 9 04:06:55 widow-oss11c2 kernel: [55141.295761] Lustre: Skipped 5 previous similar messages Mar 9 04:07:36 widow-oss11c2 kernel: [55182.648039] Lustre: routed1-OST0111: Client 4ecb0c72-92ce-ad90-881f-608038e54844 (at 16431@gni) reconnecting Mar 9 04:07:36 widow-oss11c2 kernel: [55182.673618] Lustre: Skipped 18 previous similar messages Mar 9 04:08:52 widow-oss11c2 kernel: [55258.769639] Lustre: routed1-OST00b1: Client db19d10f-56ef-0844-c0d5-58c21ecb7e6a (at 15089@gni) reconnecting Mar 9 04:08:52 widow-oss11c2 kernel: [55258.795327] Lustre: Skipped 13 previous similar messages Mar 9 04:11:12 widow-oss11c2 kernel: [55398.681402] Lustre: routed1-OST0171: Client ec80af93-ae64-b665-9c27-068087d2b3c9 (at 16492@gni) reconnecting Mar 9 04:11:12 widow-oss11c2 kernel: [55398.707349] Lustre: Skipped 25 previous similar messages Mar 9 04:13:01 widow-oss11c2 kernel: [55506.704038] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 600s: evicting client at 1003@gni ns: filter-ffff88023a903800 lock: ffff88020350a600/0x96f9706819098919 lrc: 3/0,0 mode: PW/PW res: 33297/0 rrc: 61 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 1003@gni remote: 0x2852df58071921a5 expref: 4 pid: 10244 timeout: 4350173168 lvb_type: 0 Mar 9 04:14:32 widow-oss11c2 kernel: [55597.676199] LustreError: 7501:0:(ldlm_lockd.c:480:ldlm_add_waiting_lock()) ### not waiting on destroyed lock (bug 5653) ns: filter-ffff8802e8b73000 lock: ffff88029ae9a800/0x96f97068190a5a94 lrc: 1/0,0 mode: --/PW res: 33318/0 rrc: 74 type: EXT [20480->33574911] (req 20480->40959) flags: 0x801000000020 nid: 4586@gni remote: 0x71601f4412917dff expref: 6 pid: 26568 timeout: 0 lvb_type: 0 Mar 9 04:14:32 widow-oss11c2 kernel: [55597.773177] Pid: 7501, comm: ldlm_cn01_003 Mar 9 04:14:32 widow-oss11c2 kernel: [55597.788688] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.788689] Call Trace: Mar 9 04:14:32 widow-oss11c2 kernel: [55597.810899] [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.832148] [] ldlm_add_waiting_lock+0x1cb/0x300 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.853624] [] ldlm_server_completion_ast+0x4d8/0x630 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.876039] [] ? ldlm_server_completion_ast+0x0/0x630 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.898418] [] ldlm_work_cp_ast_lock+0xcc/0x200 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.919844] [] ptlrpc_set_wait+0x6c/0x8c0 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.940165] [] ? cfs_waitq_init+0x15/0x20 [libcfs] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.960591] [] ? ptlrpc_prep_set+0xf9/0x2e0 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55597.981289] [] ? ldlm_work_cp_ast_lock+0x0/0x200 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.002777] [] ldlm_run_ast_work+0x1db/0x490 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.023651] [] ldlm_reprocess_all+0x115/0x300 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.044604] [] ldlm_request_cancel+0x277/0x410 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.065643] [] ldlm_handle_cancel+0x13d/0x240 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.086414] [] ldlm_cancel_handler+0x3f8/0x600 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.107222] [] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.129481] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.148907] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.169290] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.189489] [] ? default_wake_function+0x0/0x20 Mar 9 04:14:32 widow-oss11c2 kernel: [55598.208408] [] ptlrpc_main+0xb76/0x1870 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.227488] [] ? ptlrpc_main+0x0/0x1870 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.246486] [] child_rip+0xa/0x20 Mar 9 04:14:32 widow-oss11c2 kernel: [55598.262904] [] ? ptlrpc_main+0x0/0x1870 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.281923] [] ? ptlrpc_main+0x0/0x1870 [ptlrpc] Mar 9 04:14:32 widow-oss11c2 kernel: [55598.300756] [] ? child_rip+0x0/0x20 Mar 9 04:14:32 widow-oss11c2 kernel: [55598.317256] Mar 9 04:15:58 widow-oss11c2 kernel: [55684.434639] Lustre: routed1-OST0171: Client 455b1d95-0445-8df4-3b3d-9326e298ca52 (at 17260@gni) reconnecting Mar 9 04:15:58 widow-oss11c2 kernel: [55684.460909] Lustre: Skipped 454 previous similar messages Mar 9 04:19:17 widow-oss11c2 kernel: [55882.704338] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 420s: evicting client at 16786@gni ns: filter-ffff88023a903800 lock: ffff880268bd7400/0x96f97068190a2aac lrc: 3/0,0 mode: PW/PW res: 33297/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 16786@gni remote: 0x1915731799eca405 expref: 4 pid: 26611 timeout: 4350549103 lvb_type: 0 Mar 9 04:20:59 widow-oss11c2 kernel: [55985.704028] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 388s: evicting client at 4585@gni ns: filter-ffff8802e8b73000 lock: ffff88027a77f200/0x96f97068190a61cc lrc: 3/0,0 mode: PW/PW res: 33318/0 rrc: 3 type: EXT [0->20479] (req 0->20479) flags: 0x20 nid: 4585@gni remote: 0xefc27ad1db4c584d expref: 4 pid: 26525 timeout: 4350652624 lvb_type: 0 Mar 9 04:39:08 widow-oss11c2 kernel: [57073.704026] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 14976@gni ns: filter-ffff88039deb0800 lock: ffff8802036ec400/0x96f97068190f7541 lrc: 3/0,0 mode: PW/PW res: 34408/0 rrc: 2 type: EXT [13037568->13234175] (req 13037568->13041663) flags: 0x20 nid: 14976@gni remote: 0xc3afb2f6a69b12d8 expref: 4 pid: 10216 timeout: 4351740062 lvb_type: 0 Mar 9 04:39:08 widow-oss11c2 kernel: [57073.804521] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) Skipped 1 previous similar message Mar 9 04:46:02 widow-oss11c2 kernel: [57488.704030] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 600s: evicting client at 14845@gni ns: filter-ffff8803c3efe800 lock: ffff88029e16dc00/0x96f97068190fbe4a lrc: 3/0,0 mode: PW/PW res: 33393/0 rrc: 3 type: EXT [0->1048575] (req 0->4095) flags: 0x20 nid: 14845@gni remote: 0xf378bf7711611ae4 expref: 4 pid: 10216 timeout: 4352155469 lvb_type: 0 Mar 9 05:40:15 widow-oss11c2 kernel: [60741.704024] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 4033@gni ns: filter-ffff8803c3efe800 lock: ffff88029e068600/0x96f9706819118db6 lrc: 3/0,0 mode: PW/PW res: 35280/0 rrc: 2 type: EXT [0->4095] (req 0->4095) flags: 0x20 nid: 4033@gni remote: 0x373028a32ec59309 expref: 4 pid: 26517 timeout: 4355408284 lvb_type: 0 Mar 9 05:40:15 widow-oss11c2 kernel: [60741.801590] LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) Skipped 1 previous similar message Mar 9 06:36:20 widow-oss11c2 kernel: [64105.822452] Lustre: routed1-OST0051: haven't heard from client 1bcb657b-0596-40e3-4e5a-d55711f57870 (at 17667@gni) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802efded800, cur 1362828980 expire 1362828080 last 1362827628 Mar 9 06:36:20 widow-oss11c2 kernel: [64105.881302] Lustre: routed1-OST0051: haven't heard from client 3673cfdd-85d9-ef1e-d597-51765b4660ea (at 778@gni) in 1351 seconds. I think it's dead, and I am evicting it. exp ffff8803068a9400, cur 1362828980 expire 1362828080 last 1362827629 Mar 9 07:20:29 widow-oss11c2 kernel: [66755.450064] Lustre: 6367:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831062/real 1362831062] req@ffff880322399000 x1429008117473164/t0(0) o400->routed1-MDT0000-lwp-OST0111@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362831629 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:20:29 widow-oss11c2 kernel: [66755.450088] Lustre: routed1-MDT0000-lwp-OST0171: Connection to routed1-MDT0000 (at 10.36.227.211@o2ib) was lost; in progress operations using this service will wait for recovery to complete Mar 9 07:20:29 widow-oss11c2 kernel: [66755.581759] Lustre: 6367:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 9 07:20:29 widow-oss11c2 kernel: [66755.609612] Lustre: routed1-MDT0000-lwp-OST0111: Connection to routed1-MDT0000 (at 10.36.227.211@o2ib) was lost; in progress operations using this service will wait for recovery to complete Mar 9 07:20:29 widow-oss11c2 kernel: [66755.660132] Lustre: Skipped 4 previous similar messages Mar 9 07:22:59 widow-oss11c2 kernel: [66905.450033] Lustre: 6364:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831212/real 1362831212] req@ffff88031663b000 x1429008117473184/t0(0) o400->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362831779 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:22:59 widow-oss11c2 kernel: [66905.450040] Lustre: 6366:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831212/real 1362831212] req@ffff880270fc6400 x1429008117473192/t0(0) o400->routed1-MDT0000-lwp-OST0111@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362831779 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:22:59 widow-oss11c2 kernel: [66905.450046] Lustre: 6366:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 9 07:22:59 widow-oss11c2 kernel: [66905.646697] Lustre: 6364:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 9 07:25:09 widow-oss11c2 kernel: [67035.680035] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831629/real 1362831629] req@ffff88022318f400 x1429008117473284/t0(0) o38->routed1-MDT0000-lwp-OST0111@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362831909 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:25:29 widow-oss11c2 kernel: [67055.450028] Lustre: 6366:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831362/real 1362831362] req@ffff88033a935400 x1429008117473216/t0(0) o400->routed1-MDT0000-lwp-OST00b1@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362831929 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:25:29 widow-oss11c2 kernel: [67055.535967] Lustre: 6366:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 9 07:27:59 widow-oss11c2 kernel: [67205.451030] Lustre: 6368:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831512/real 1362831512] req@ffff880204308800 x1429008117473256/t0(0) o400->routed1-MDT0000-lwp-OST01d1@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362832079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:27:59 widow-oss11c2 kernel: [67205.537156] Lustre: 6368:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 9 07:30:09 widow-oss11c2 kernel: [67335.679028] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362831929/real 1362831929] req@ffff8802d8c0c400 x1429008117473316/t0(0) o38->routed1-MDT0000-lwp-OST0231@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362832209 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:30:09 widow-oss11c2 kernel: [67335.764026] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 9 07:30:42 widow-oss11c2 kernel: [67367.875762] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1362832229/real 1362832242] req@ffff88022318f400 x1429008117473328/t0(0) o38->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362832509 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:30:42 widow-oss11c2 kernel: [67367.961175] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 9 07:34:39 widow-oss11c2 kernel: [67605.683073] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1362832379/real 1362832479] req@ffff8802db73b400 x1429008117473356/t0(0) o38->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362832659 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:34:39 widow-oss11c2 kernel: [67605.768568] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 9 07:36:49 widow-oss11c2 kernel: [67734.766395] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1362832529/real 1362832608] req@ffff8803ea4b0800 x1429008117473384/t0(0) o38->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362832809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:36:49 widow-oss11c2 kernel: [67734.851314] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 9 07:45:09 widow-oss11c2 kernel: [68235.684031] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362832829/real 1362832829] req@ffff8803bc5c3400 x1429008117473428/t0(0) o38->routed1-MDT0000-lwp-OST0111@10.36.227.211@o2ib:12/10 lens 400/544 e 0 to 1 dl 1362833109 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 07:45:09 widow-oss11c2 kernel: [68235.767701] Lustre: 6361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 9 07:45:29 widow-oss11c2 kernel: [68255.679416] LustreError: 167-0: routed1-MDT0000-lwp-OST00b1: This client was evicted by routed1-MDT0000; in progress operations using this service will fail. Mar 9 07:45:29 widow-oss11c2 kernel: [68255.724950] LustreError: 167-0: routed1-MDT0000-lwp-OST0111: This client was evicted by routed1-MDT0000; in progress operations using this service will fail. Mar 9 07:45:29 widow-oss11c2 kernel: [68255.725742] Lustre: routed1-MDT0000-lwp-OST00b1: Connection restored to routed1-MDT0000 (at 10.36.227.211@o2ib) Mar 9 07:45:30 widow-oss11c2 kernel: [68255.800426] Lustre: routed1-MDT0000-lwp-OST0171: Connection restored to routed1-MDT0000 (at 10.36.227.211@o2ib) Mar 9 07:45:30 widow-oss11c2 kernel: [68255.829256] Lustre: Skipped 4 previous similar messages Mar 9 07:45:56 widow-oss11c2 kernel: [68282.383239] Lustre: routed1-OST0051: deleting orphan objects from 0x0:37003 to 37024 Mar 9 07:45:56 widow-oss11c2 kernel: [68282.407471] Lustre: Skipped 3 previous similar messages Mar 9 09:00:05 widow-oss11c2 kernel: [72731.299457] Lustre: routed1-OST0111: Client 517359b1-794e-ad51-3df0-ec7359e95a19 (at 3024@gni) reconnecting Mar 9 09:00:05 widow-oss11c2 kernel: [72731.327612] Lustre: Skipped 35 previous similar messages Mar 9 09:01:15 widow-oss11c2 kernel: [72801.255795] Lustre: routed1-OST0051: Client ae0d3490-a93d-5605-e1fe-1ec673244983 (at 16066@gni) reconnecting Mar 9 09:01:15 widow-oss11c2 kernel: [72801.284026] Lustre: Skipped 7805 previous similar messages Mar 9 09:01:27 widow-oss11c2 kernel: [72813.716024] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 2 seconds Mar 9 09:01:27 widow-oss11c2 kernel: [72813.717004] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.204@o2ib (102): c: 0, oc: 0, rc: 8 Mar 9 09:01:42 widow-oss11c2 kernel: [72828.716024] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 9 09:01:42 widow-oss11c2 kernel: [72828.717007] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.61@o2ib (101): c: 0, oc: 0, rc: 8 Mar 9 09:01:45 widow-oss11c2 kernel: [72831.716050] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Mar 9 09:01:45 widow-oss11c2 kernel: [72831.717022] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.73@o2ib (102): c: 0, oc: 0, rc: 8 Mar 9 09:01:48 widow-oss11c2 kernel: [72833.716022] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Mar 9 09:01:48 widow-oss11c2 kernel: [72833.717007] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Mar 9 09:01:48 widow-oss11c2 kernel: [72833.717007] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.80@o2ib (102): c: 0, oc: 0, rc: 8 Mar 9 09:01:48 widow-oss11c2 kernel: [72833.717007] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Mar 9 09:01:51 widow-oss11c2 kernel: [72836.716023] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 5 seconds Mar 9 09:01:51 widow-oss11c2 kernel: [72836.717005] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Mar 9 09:01:51 widow-oss11c2 kernel: [72836.717005] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.92@o2ib (105): c: 0, oc: 0, rc: 8 Mar 9 09:01:51 widow-oss11c2 kernel: [72836.717005] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Mar 9 09:01:56 widow-oss11c2 kernel: [72841.716029] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 10 seconds Mar 9 09:01:56 widow-oss11c2 kernel: [72841.717006] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 10 previous similar messages Mar 9 09:01:56 widow-oss11c2 kernel: [72841.717006] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.13@o2ib (110): c: 0, oc: 0, rc: 8 Mar 9 09:01:56 widow-oss11c2 kernel: [72841.717006] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 10 previous similar messages Mar 9 09:02:05 widow-oss11c2 kernel: [72850.716023] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds Mar 9 09:02:05 widow-oss11c2 kernel: [72850.717006] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 15 previous similar messages Mar 9 09:02:05 widow-oss11c2 kernel: [72850.717006] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.49@o2ib (120): c: 0, oc: 0, rc: 8 Mar 9 09:02:05 widow-oss11c2 kernel: [72850.717006] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 15 previous similar messages Mar 9 09:02:22 widow-oss11c2 kernel: [72867.716036] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 25 seconds Mar 9 09:02:22 widow-oss11c2 kernel: [72867.717019] LNetError: 5687:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 18 previous similar messages Mar 9 09:02:22 widow-oss11c2 kernel: [72867.717019] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.229.115@o2ib (125): c: 0, oc: 0, rc: 8 Mar 9 09:02:22 widow-oss11c2 kernel: [72867.717019] LNetError: 5687:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 18 previous similar messages Mar 9 09:03:23 widow-oss11c2 kernel: [72929.270752] Lustre: routed1-OST0111: Client a65900bb-57f4-0103-c7bb-deb0296aeb96 (at 1389@gni) reconnecting Mar 9 09:03:23 widow-oss11c2 kernel: [72929.298295] Lustre: Skipped 6140 previous similar messages Mar 9 09:10:54 widow-oss11c2 kernel: [73380.488102] Lustre: 22238:0:(service.c:1296:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Mar 9 09:10:54 widow-oss11c2 kernel: [73380.488104] req@ffff8801ea442050 x1428977289665852/t0(0) o4->20644ff1-5fbb-ce88-11ac-81aebe565653@82@gni:0/0 lens 488/448 e 1 to 0 dl 1362838259 ref 2 fl Interpret:/0/0 rc 0/0 Mar 9 09:10:59 widow-oss11c2 kernel: [73385.491034] LustreError: 22264:0:(ldlm_lib.c:2705:target_bulk_io()) @@@ timeout on bulk GET after 654+0s req@ffff8801ea442050 x1428977289665852/t0(0) o4->20644ff1-5fbb-ce88-11ac-81aebe565653@82@gni:0/0 lens 488/448 e 1 to 0 dl 1362838259 ref 1 fl Interpret:/0/0 rc 0/0 Mar 9 09:10:59 widow-oss11c2 kernel: [73385.562742] Lustre: routed1-OST0231: Bulk IO write error with 20644ff1-5fbb-ce88-11ac-81aebe565653 (at 82@gni), client will retry: rc -110 Mar 9 09:19:51 widow-oss11c2 kernel: [73917.426854] Lustre: routed1-OST01d1: haven't heard from client 011ff8e6-aabf-5f2e-b9eb-c3a497043e9e (at 2040@gni) in 1357 seconds. I think it's dead, and I am evicting it. exp ffff8802bd320800, cur 1362838791 expire 1362837891 last 1362837434 Mar 9 09:19:51 widow-oss11c2 kernel: [73917.487171] Lustre: Skipped 10 previous similar messages Mar 9 09:19:51 widow-oss11c2 kernel: [73917.506769] Lustre: routed1-OST01d1: haven't heard from client 3d3a41d4-58c2-6a33-d0cc-6608671289b3 (at 1364@gni) in 1356 seconds. I think it's dead, and I am evicting it. exp ffff88042d1a5800, cur 1362838791 expire 1362837891 last 1362837435 Mar 9 09:19:52 widow-oss11c2 kernel: [73918.008024] Lustre: routed1-OST0171: haven't heard from client 42f79fb7-33d8-de16-ccd8-9f53b5a84146 (at 19147@gni) in 1005 seconds. I think it's dead, and I am evicting it. exp ffff880294fa4c00, cur 1362838792 expire 1362837892 last 1362837787 Mar 9 09:19:52 widow-oss11c2 kernel: [73918.068916] Lustre: Skipped 15171 previous similar messages Mar 9 09:19:56 widow-oss11c2 kernel: [73922.425669] Lustre: routed1-OST0111: haven't heard from client 8da5c87c-2a03-8e44-da61-11327ed29117 (at 18500@gni) in 1356 seconds. I think it's dead, and I am evicting it. exp ffff880293827c00, cur 1362838796 expire 1362837896 last 1362837440 Mar 9 09:19:56 widow-oss11c2 kernel: [73922.486586] Lustre: Skipped 29716 previous similar messages Mar 9 12:02:11 widow-oss11c2 kernel: [83657.489117] LNet: No route to 12345-1857@gni via 10.36.227.130@o2ib (all routers down) Mar 9 12:02:11 widow-oss11c2 kernel: [83657.514421] LNet: No route to 12345-17221@gni via 10.36.227.130@o2ib (all routers down) Mar 9 12:09:07 widow-oss11c2 kernel: [84073.305741] Lustre: routed1-OST0111: Client faa236fc-db46-1722-f8a2-cb7e1260c022 (at 707@gni) reconnecting Mar 9 12:09:07 widow-oss11c2 kernel: [84073.334215] Lustre: Skipped 7568 previous similar messages Mar 9 12:33:51 widow-oss11c2 kernel: Kernel logging (proc) stopped. Mar 9 12:33:51 widow-oss11c2 kernel: imklog 5.8.10, log source = /proc/kmsg started. Mar 9 14:06:58 widow-oss11c2 kernel: [91144.011729] LustreError: 10226:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 211812352 != fo_tot_granted 213909504 Mar 9 14:06:58 widow-oss11c2 kernel: [91144.395097] LustreError: 26508:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 213909504 != fo_tot_granted 216006656 Mar 9 14:06:58 widow-oss11c2 kernel: [91144.439318] LustreError: 26508:0:(ofd_grant.c:163:ofd_grant_sanity_check()) Skipped 1 previous similar message Mar 9 14:06:59 widow-oss11c2 kernel: [91144.983144] LustreError: 10228:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 62916096 != fo_tot_granted 65013248 Mar 9 14:07:00 widow-oss11c2 kernel: [91146.275611] LustreError: 16920:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 82837504 != fo_tot_granted 84934656 Mar 9 20:11:11 widow-oss11c2 kernel: [112997.034048] Lustre: 20361:0:(client.c:1866:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1362877591/real 1362877591] req@ffff8802ee6d5400 x1429008118270248/t0(0) o39->routed1-MDT0000-lwp-OST0051@10.36.227.211@o2ib:12/10 lens 224/224 e 0 to 1 dl 1362877871 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 20:11:11 widow-oss11c2 kernel: [112997.120230] Lustre: 20361:0:(client.c:1866:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 9 20:11:11 widow-oss11c2 kernel: [112997.150249] LustreError: 20361:0:(obd_mount.c:2115:server_put_super()) routed1-OST0051: failed to disconnect lwp. (rc=-110) Mar 9 20:11:11 widow-oss11c2 kernel: [112997.182466] Lustre: Failing over routed1-OST0051 Mar 9 20:11:11 widow-oss11c2 kernel: [112997.435784] LustreError: 20402:0:(obd_mount.c:2115:server_put_super()) routed1-OST00b1: failed to disconnect lwp. (rc=-110) Mar 9 20:11:11 widow-oss11c2 kernel: [112997.468118] Lustre: Failing over routed1-OST00b1 Mar 9 20:11:12 widow-oss11c2 kernel: [112997.838378] Lustre: server umount routed1-OST0051 complete Mar 9 20:11:12 widow-oss11c2 kernel: [112998.001774] LustreError: 20451:0:(obd_mount.c:2115:server_put_super()) routed1-OST0111: failed to disconnect lwp. (rc=-110) Mar 9 20:11:12 widow-oss11c2 kernel: [112998.034264] Lustre: Failing over routed1-OST0111 Mar 9 20:11:12 widow-oss11c2 kernel: [112998.231220] Lustre: server umount routed1-OST00b1 complete Mar 9 20:11:13 widow-oss11c2 kernel: [112999.326796] LustreError: 20552:0:(obd_mount.c:2115:server_put_super()) routed1-OST01d1: failed to disconnect lwp. (rc=-110) Mar 9 20:11:13 widow-oss11c2 kernel: [112999.359171] LustreError: 20552:0:(obd_mount.c:2115:server_put_super()) Skipped 1 previous similar message Mar 9 20:11:13 widow-oss11c2 kernel: [112999.370669] Lustre: server umount routed1-OST0171 complete Mar 9 20:11:13 widow-oss11c2 kernel: [112999.370672] Lustre: Skipped 1 previous similar message Mar 9 20:11:13 widow-oss11c2 kernel: [112999.429394] Lustre: Failing over routed1-OST01d1 Mar 9 20:11:13 widow-oss11c2 kernel: [112999.448337] Lustre: Skipped 1 previous similar message Mar 9 20:12:30 widow-oss11c2 kernel: [113076.073001] BUG: soft lockup - CPU#0 stuck for 67s! [obd_zombid:5676] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.073001] Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ext4 mbcache jbd2 sha512_generic sha256_generic ipmi_devintf ipmi_si ipmi_msghandler autofs4 dm_round_robin scsi_wait_scan sd_mod crc_t10dif ib_srp scsi_transport_srp scsi_tgt nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sr_mod cdrom radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core ib_mthca ib_mad ib_core shpchp microcode dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support i5000_edac edac_core i5k_amb ioatdma dca ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc bnx2 [last unloaded: libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] CPU 0 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ext4 mbcache jbd2 sha512_generic sha256_generic ipmi_devintf ipmi_si ipmi_msghandler autofs4 dm_round_robin scsi_wait_scan sd_mod crc_t10dif ib_srp scsi_transport_srp scsi_tgt nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sr_mod cdrom radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core ib_mthca ib_mad ib_core shpchp microcode dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support i5000_edac edac_core i5k_amb ioatdma dca ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc bnx2 [last unloaded: libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Pid: 5676, comm: obd_zombid Not tainted 2.6.32-279.22.1.el6.head.x86_64 #1 Dell Inc. PowerEdge 1950/0TT740 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] RIP: 0010:[] [] remove_vm_area+0x86/0xa0 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] RSP: 0018:ffff880419427cb0 EFLAGS: 00000287 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] RAX: ffff8800a66fb540 RBX: ffff880419427cd0 RCX: ffff88011e073440 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] RDX: ffff88011e0733c0 RSI: 0000000000000001 RDI: ffffffff81abea30 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] RBP: ffffffff8100bb8e R08: 0000000000000000 R09: ffffc900cc6ee000 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] R10: ffff880028402500 R11: ffff88008acf3800 R12: ffff880419427c80 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] R13: 0000000000000297 R14: ffff8800000206c0 R15: ffffea0002410270 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] FS: 00007f4834066700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] CR2: 00000000004050b9 CR3: 0000000001a85000 CR4: 00000000000006f0 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Process obd_zombid (pid: 5676, threadinfo ffff880419426000, task ffff8803e2329540) Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Stack: Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] ffff88001b6df840 ffff8800a66fb540 ffff88008ad17850 ffffc900cc6ee000 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] ffff880419427d00 ffffffff81147f4e 000000000000005a ffff88006f537800 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] ffff8803d5e93400 ffff8803d5e93430 ffff880419427d10 ffffffff811480ba Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Call Trace: Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? __vunmap+0x2e/0x120 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? vfree+0x2a/0x40 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? cfs_free_large+0xe/0x10 [libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ptlrpc_free_rqbd+0xc2/0x3b0 [ptlrpc] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ptlrpc_unregister_service+0x5d5/0x10b0 [ptlrpc] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? __wake_up+0x53/0x70 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ost_cleanup+0x52/0x1a0 [ost] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? class_decref+0x212/0x580 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_cull+0x309/0x5d0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0xc5/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? default_wake_function+0x0/0x20 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? child_rip+0xa/0x20 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? child_rip+0x0/0x20 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Code: 30 ea ab 81 48 89 45 e8 e8 88 4d 3a 00 48 8b 15 19 73 e6 00 48 c7 c1 78 f1 fa 81 48 8b 45 e8 48 39 d3 74 0c 90 48 89 d1 48 8b 12 <48> 39 d3 75 f5 48 8b 13 48 89 11 f0 81 05 a4 6b 97 00 00 00 00 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] Call Trace: Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? __vunmap+0x2e/0x120 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? vfree+0x2a/0x40 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? cfs_free_large+0xe/0x10 [libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ptlrpc_free_rqbd+0xc2/0x3b0 [ptlrpc] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ptlrpc_unregister_service+0x5d5/0x10b0 [ptlrpc] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? __wake_up+0x53/0x70 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? ost_cleanup+0x52/0x1a0 [ost] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? class_decref+0x212/0x580 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_cull+0x309/0x5d0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0xc5/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? default_wake_function+0x0/0x20 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? child_rip+0xa/0x20 Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] Mar 9 20:12:30 widow-oss11c2 kernel: [113076.102203] [] ? child_rip+0x0/0x20 Mar 9 20:13:35 widow-oss11c2 kernel: [113140.803317] Lustre: server umount routed1-OST0231 complete Mar 9 20:13:35 widow-oss11c2 kernel: [113140.819063] Lustre: Skipped 1 previous similar message