Nov 18 11:42:18 widow-oss13c2 kernel: [1725442.686864] LustreError: 5003:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0069_UUID: lvbo_init failed for resource 21568871: rc -2 Nov 18 11:42:18 widow-oss14a4 kernel: [1725402.123574] LustreError: 16258:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST006f_UUID: lvbo_init failed for resource 21898851: rc -2 Nov 18 11:42:18 widow-oss16a2 kernel: [1725403.530444] LustreError: 4446:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0025_UUID: lvbo_init failed for resource 23848791: rc -2 Nov 18 11:42:18 widow-oss13c2 kernel: [1725442.843936] LustreError: 5025:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST00f9_UUID: lvbo_init failed for resource 24056285: rc -2 Nov 18 11:42:18 widow-oss15b4 kernel: [1725406.372648] LustreError: 17473:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST00af_UUID: lvbo_init failed for resource 22878123: rc -2 Nov 18 11:42:18 widow-oss15a3 kernel: [1725379.794020] LustreError: 17411:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST013a_UUID: lvbo_init failed for resource 23797266: rc -2 Nov 18 11:42:18 widow-oss13c1 kernel: [1725415.138880] LustreError: 14420:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0038_UUID: lvbo_init failed for resource 23397643: rc -2 Nov 18 11:42:18 widow-oss14b3 kernel: [1725422.319417] LustreError: 16147:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0132_UUID: lvbo_init failed for resource 22260358: rc -2 Nov 18 11:42:18 widow-oss15a4 kernel: [1725410.867092] LustreError: 17317:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST007b_UUID: lvbo_init failed for resource 22800472: rc -2 Nov 18 11:42:18 widow-oss16a1 kernel: [1725426.810475] LustreError: 18440:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST00e4_UUID: lvbo_init failed for resource 23743242: rc -2 Nov 18 11:42:18 widow-oss16a4 kernel: [1725455.495592] LustreError: 6226:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0087_UUID: lvbo_init failed for resource 23422346: rc -2 Nov 18 11:42:18 widow-oss15c3 kernel: [1725440.367622] LustreError: 17378:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0022_UUID: lvbo_init failed for resource 24014948: rc -2 Nov 18 11:42:18 widow-oss14c3 kernel: [1725387.313634] LustreError: 21636:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0136_UUID: lvbo_init failed for resource 23089457: rc -2 Nov 18 11:42:18 widow-oss15a4 kernel: [1725410.931988] LustreError: 10727:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST013b_UUID: lvbo_init failed for resource 23003980: rc -2 Nov 18 11:42:18 widow-oss15c1 kernel: [1725436.844270] LustreError: 17410:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST00e0_UUID: lvbo_init failed for resource 23496705: rc -2 Nov 18 11:42:18 widow-oss16b1 kernel: [1725428.466604] LustreError: 18368:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0058_UUID: lvbo_init failed for resource 23406313: rc -2 Nov 18 11:42:19 widow-oss13a1 kernel: [1725410.792362] LustreError: 17469:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0060_UUID: lvbo_init failed for resource 23015364: rc -2 Nov 18 11:42:19 widow-oss16a2 kernel: [1725404.358367] LustreError: 16169:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0145_UUID: lvbo_init failed for resource 22057549: rc -2 Nov 18 11:42:19 widow-oss14c1 kernel: [1725415.860585] LustreError: 15929:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0014_UUID: lvbo_init failed for resource 23017250: rc -2 Nov 18 11:42:19 widow-oss15b2 kernel: [1725440.333898] LustreError: 7915:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST00dd_UUID: lvbo_init failed for resource 23527375: rc -2 Nov 18 11:42:19 widow-oss13c2 kernel: [1725443.630272] LustreError: 4999:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow3-OST0039_UUID: lvbo_init failed for resource 22696073: rc -2 Nov 18 11:46:10 widow-mds2 kernel: [1726316.709869] LustreError: 9088:0:(handler.c:2515:mds_intent_policy()) ASSERTION(new_lock != NULL) failed: op 0x8 lockh 0x0 Nov 18 11:46:10 widow-mds2 kernel: [1726316.732069] LustreError: 9088:0:(handler.c:2515:mds_intent_policy()) LBUG Nov 18 11:46:10 widow-mds2 kernel: [1726316.745957] Pid: 9088, comm: ll_mdt_73 Nov 18 11:46:10 widow-mds2 kernel: [1726316.745959] Nov 18 11:46:10 widow-mds2 kernel: [1726316.745960] Call Trace: Nov 18 11:46:10 widow-mds2 kernel: [1726316.746063] [<ffffffff887bb74f>] libcfs_debug_dumpstack+0x5f/0x80 [libcfs] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746082] [<ffffffff887bbc7f>] lbug_with_loc+0x7f/0xd0 [libcfs] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746128] [<ffffffff88c0162d>] mds_intent_policy+0x8ad/0xbd0 [mds] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746227] [<ffffffff88912f2a>] ldlm_lock_enqueue+0x17a/0xb00 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746261] [<ffffffff8891580b>] ldlm_resource_putref+0xb/0x10 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746289] [<ffffffff8890f80c>] ldlm_lock_create+0x99c/0x9d0 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746351] [<ffffffff8888f02d>] lustre_hash_add+0x3bd/0x3e0 [obdclass] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746394] [<ffffffff88937620>] ldlm_server_blocking_ast+0x0/0x847 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746424] [<ffffffff8893490b>] ldlm_handle_enqueue+0xbfb/0x1200 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746453] [<ffffffff88c0014f>] mds_handle+0x409f/0x4cd0 [mds] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746468] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:46:10 widow-mds2 kernel: [1726316.746519] [<ffffffff88958f45>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746540] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:46:10 widow-mds2 kernel: [1726316.746573] [<ffffffff889530f5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746606] [<ffffffff88962940>] ptlrpc_server_handle_request+0x980/0xdd0 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746636] [<ffffffff88963098>] ptlrpc_wait_event+0x308/0x320 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746673] [<ffffffff88963ff6>] ptlrpc_main+0xf46/0x1110 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746685] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:46:10 widow-mds2 kernel: [1726316.746698] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:46:10 widow-mds2 kernel: [1726316.746733] [<ffffffff889630b0>] ptlrpc_main+0x0/0x1110 [ptlrpc] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746738] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:46:10 widow-mds2 kernel: [1726316.746743] Nov 18 11:46:10 widow-mds2 kernel: [1726316.746981] LustreError: dumping log to /tmp/lustre-log.1384793170.9088 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147729] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:48:15 widow-mds2 kernel: [1726441.147736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:48:15 widow-mds2 kernel: [1726441.147739] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:48:15 widow-mds2 kernel: [1726441.147746] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147759] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147775] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147785] Call Trace: Nov 18 11:48:15 widow-mds2 kernel: [1726441.147835] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:48:15 widow-mds2 kernel: [1726441.147858] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147890] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147954] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:48:15 widow-mds2 kernel: [1726441.147987] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148020] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148062] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148099] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148129] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148157] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148165] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148210] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148219] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148258] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148297] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148336] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148382] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148396] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148409] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148465] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148473] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:48:15 widow-mds2 kernel: [1726441.148479] Nov 18 11:50:15 widow-mds2 kernel: [1726560.935188] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:50:15 widow-mds2 kernel: [1726560.935194] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:50:15 widow-mds2 kernel: [1726560.935197] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:50:15 widow-mds2 kernel: [1726560.935206] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935221] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935247] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935263] Call Trace: Nov 18 11:50:15 widow-mds2 kernel: [1726560.935321] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:50:15 widow-mds2 kernel: [1726560.935349] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935377] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935443] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935477] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935510] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935550] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935587] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935619] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935649] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935657] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935704] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935715] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935753] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935789] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935821] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935862] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935876] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935889] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935935] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935960] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:50:15 widow-mds2 kernel: [1726560.935971] Nov 18 11:52:15 widow-mds2 kernel: [1726680.722689] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:52:15 widow-mds2 kernel: [1726680.722696] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:52:15 widow-mds2 kernel: [1726680.722699] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:52:15 widow-mds2 kernel: [1726680.722711] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722722] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722734] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722745] Call Trace: Nov 18 11:52:15 widow-mds2 kernel: [1726680.722798] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:52:15 widow-mds2 kernel: [1726680.722821] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722857] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722932] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:52:15 widow-mds2 kernel: [1726680.722967] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723000] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723056] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723104] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723139] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723177] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723188] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723234] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723245] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723279] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723314] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723352] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723400] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723428] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723445] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723480] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723485] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:52:15 widow-mds2 kernel: [1726680.723493] Nov 18 11:53:15 widow-oss12c3 kernel: [1726076.356014] LustreError: 6946:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST008e_UUID: lvbo_init failed for resource 12841552: rc -2 Nov 18 11:53:15 widow-oss9a1 kernel: [1726071.447526] LustreError: 18681:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST0090_UUID: lvbo_init failed for resource 12764583: rc -2 Nov 18 11:53:15 widow-oss12c4 kernel: [1726037.512934] LustreError: 15258:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST008f_UUID: lvbo_init failed for resource 12691760: rc -2 Nov 18 11:53:15 widow-oss9a2 kernel: [1726047.320551] LustreError: 17817:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST0091_UUID: lvbo_init failed for resource 12513590: rc -2 Nov 18 11:53:37 widow-oss11a2 kernel: [1726111.683095] LustreError: 10445:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST00d9_UUID: lvbo_init failed for resource 12625186: rc -2 Nov 18 11:53:37 widow-oss10c4 kernel: [1726115.054860] LustreError: 15610:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST00d7_UUID: lvbo_init failed for resource 12824919: rc -2 Nov 18 11:53:37 widow-oss11a1 kernel: [1726100.105972] LustreError: 11363:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST00d8_UUID: lvbo_init failed for resource 12890514: rc -2 Nov 18 11:53:37 widow-oss10c3 kernel: [1726075.188817] LustreError: 9970:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST00d6_UUID: lvbo_init failed for resource 12784926: rc -2 Nov 18 11:53:37 widow-oss9a1 kernel: [1726094.208905] LustreError: 17356:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST0120_UUID: lvbo_init failed for resource 12498228: rc -2 Nov 18 11:53:37 widow-oss9a2 kernel: [1726070.081609] LustreError: 17746:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST0121_UUID: lvbo_init failed for resource 12404183: rc -2 Nov 18 11:53:37 widow-oss12c3 kernel: [1726099.117473] LustreError: 14719:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST011e_UUID: lvbo_init failed for resource 12435321: rc -2 Nov 18 11:53:37 widow-oss12c4 kernel: [1726060.273879] LustreError: 6148:0:(ldlm_resource.c:864:ldlm_resource_add()) filter-widow1-OST011f_UUID: lvbo_init failed for resource 12864227: rc -2 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510161] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:54:15 widow-mds2 kernel: [1726800.510168] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:54:15 widow-mds2 kernel: [1726800.510170] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:54:15 widow-mds2 kernel: [1726800.510179] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510196] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510208] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510223] Call Trace: Nov 18 11:54:15 widow-mds2 kernel: [1726800.510274] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:54:15 widow-mds2 kernel: [1726800.510297] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510333] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510399] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510431] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510463] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510505] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510542] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510580] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510610] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510622] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510668] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510678] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510715] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510755] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510785] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510827] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510842] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510855] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510905] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510932] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:54:15 widow-mds2 kernel: [1726800.510952] Nov 18 11:56:15 widow-mds2 kernel: [1726920.297652] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:56:15 widow-mds2 kernel: [1726920.297658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:56:15 widow-mds2 kernel: [1726920.297661] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:56:15 widow-mds2 kernel: [1726920.297671] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297685] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297699] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297710] Call Trace: Nov 18 11:56:15 widow-mds2 kernel: [1726920.297765] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:56:15 widow-mds2 kernel: [1726920.297788] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297817] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297887] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297924] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:56:15 widow-mds2 kernel: [1726920.297958] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298001] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298037] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298076] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298102] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298113] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298156] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298167] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298200] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298232] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298267] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298308] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298323] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298337] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298385] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298395] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:56:15 widow-mds2 kernel: [1726920.298400] Nov 18 11:56:59 widow-mds2 kernel: [1726964.709247] Lustre: 9066:0:(service.c:812:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Nov 18 11:56:59 widow-mds2 kernel: [1726964.709250] req@ffff810bf5b73400 x1451788436424268/t0 o101->e90b3a65-adba-15cd-b725-b82e8cdc8397@NET_0xd006400002135_UUID:0/0 lens 552/480 e 1 to 0 dl 1384793824 ref 2 fl Interpret:/0/0 rc 0/0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085105] INFO: task ll_mdt_73:9088 blocked for more than 120 seconds. Nov 18 11:58:15 widow-mds2 kernel: [1727040.085112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 18 11:58:15 widow-mds2 kernel: [1727040.085114] ll_mdt_73 D ffff810009026e20 0 9088 1 9089 9087 (L-TLB) Nov 18 11:58:15 widow-mds2 kernel: [1727040.085122] ffff810e1d975a90 0000000000000046 ffff810e1d9759f0 ffff810e1d975a50 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085138] 00000000000009d3 0000000000000009 ffff810e21bff0c0 ffff810138fcf080 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085154] 000622137084d8cd 000000000000202e ffff810e21bff2a8 00000004000009d3 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085165] Call Trace: Nov 18 11:58:15 widow-mds2 kernel: [1727040.085218] [<ffffffff80094cf8>] default_wake_function+0x0/0xf Nov 18 11:58:15 widow-mds2 kernel: [1727040.085247] [<ffffffff887bbcca>] :libcfs:lbug_with_loc+0xca/0xd0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085276] [<ffffffff88c0162d>] :mds:mds_intent_policy+0x8ad/0xbd0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085343] [<ffffffff88912f2a>] :ptlrpc:ldlm_lock_enqueue+0x17a/0xb00 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085376] [<ffffffff8891580b>] :ptlrpc:ldlm_resource_putref+0xb/0x10 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085410] [<ffffffff8890f80c>] :ptlrpc:ldlm_lock_create+0x99c/0x9d0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085448] [<ffffffff8888f02d>] :obdclass:lustre_hash_add+0x3bd/0x3e0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085488] [<ffffffff88937620>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x847 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085518] [<ffffffff8893490b>] :ptlrpc:ldlm_handle_enqueue+0xbfb/0x1200 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085557] [<ffffffff88c0014f>] :mds:mds_handle+0x409f/0x4cd0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085567] [<ffffffff8007c693>] smp_send_reschedule+0x4a/0x50 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085610] [<ffffffff88958f45>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085626] [<ffffffff80049582>] try_to_wake_up+0x472/0x484 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085663] [<ffffffff889530f5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085708] [<ffffffff88962940>] :ptlrpc:ptlrpc_server_handle_request+0x980/0xdd0 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085741] [<ffffffff88963098>] :ptlrpc:ptlrpc_wait_event+0x308/0x320 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085781] [<ffffffff88963ff6>] :ptlrpc:ptlrpc_main+0xf46/0x1110 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085800] [<ffffffff800c28ad>] audit_syscall_exit+0x32b/0x346 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085817] [<ffffffff80061fc1>] child_rip+0xa/0x11 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085861] [<ffffffff889630b0>] :ptlrpc:ptlrpc_main+0x0/0x1110 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085874] [<ffffffff80061fb7>] child_rip+0x0/0x11 Nov 18 11:58:15 widow-mds2 kernel: [1727040.085879]