Jun 7 18:40:41 soak-11 sssd_be: GSSAPI client step 1 Jun 7 18:40:41 soak-11 sssd_be: GSSAPI client step 1 Jun 7 18:40:41 soak-11 sssd_be: GSSAPI client step 1 Jun 7 18:40:41 soak-11 sssd_be: GSSAPI client step 2 Jun 7 18:42:00 soak-11 kernel: Lustre: 4698:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1496860913/real 0] req@ffff8807ec723c00 x1569566643161472/t0(0) o13->soaked-OST0011-osc-MDT0003@192.168.1.107@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1496860920 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 7 18:42:00 soak-11 kernel: Lustre: soaked-OST0011-osc-MDT0003: Connection to soaked-OST0011 (at 192.168.1.107@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:42:01 soak-11 kernel: Lustre: soaked-OST0005-osc-MDT0003: Connection to soaked-OST0005 (at 192.168.1.107@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:42:01 soak-11 kernel: Lustre: Skipped 1 previous similar message Jun 7 18:42:08 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1496860922/real 0] req@ffff8807f5025700 x1569566643175424/t0(0) o8->soaked-OST000b-osc-MDT0003@192.168.1.107@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1496860928 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 7 18:42:08 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Jun 7 18:42:52 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496860972/real 1496860972] req@ffff8807d17afb00 x1569566644087696/t0(0) o8->soaked-OST0005-osc-MDT0003@192.168.1.107@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1496860983 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 7 18:43:42 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496861022/real 1496861022] req@ffff88034ef54500 x1569566645514128/t0(0) o8->soaked-OST0011-osc-MDT0003@192.168.1.107@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1496861038 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 7 18:43:42 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 7 18:45:22 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496861122/real 1496861122] req@ffff8807ff78ad00 x1569566648568976/t0(0) o8->soaked-OST000b-osc-MDT0003@192.168.1.107@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1496861148 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 7 18:45:22 soak-11 kernel: Lustre: 4675:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jun 7 18:46:08 soak-11 kernel: Lustre: soaked-MDT0003: Received LWP connection from 192.168.1.106@o2ib10, removing former export from 192.168.1.107@o2ib10 Jun 7 18:46:08 soak-11 kernel: Lustre: soaked-MDT0003: Connection restored to 192.168.1.106@o2ib10 (at 192.168.1.106@o2ib10) Jun 7 18:46:17 soak-11 kernel: Lustre: soaked-MDT0003: Received LWP connection from 192.168.1.106@o2ib10, removing former export from 192.168.1.107@o2ib10 Jun 7 18:46:25 soak-11 kernel: Lustre: soaked-MDT0003: Received LWP connection from 192.168.1.106@o2ib10, removing former export from 192.168.1.107@o2ib10 Jun 7 18:46:25 soak-11 kernel: Lustre: soaked-MDT0003: Connection restored to 192.168.1.106@o2ib10 (at 192.168.1.106@o2ib10) Jun 7 18:46:25 soak-11 kernel: Lustre: Skipped 1 previous similar message Jun 7 18:46:34 soak-11 kernel: Lustre: soaked-MDT0003: Received LWP connection from 192.168.1.106@o2ib10, removing former export from 192.168.1.107@o2ib10 Jun 7 18:47:07 soak-11 kernel: LustreError: 11-0: soaked-OST0017-osc-MDT0003: operation ost_destroy to node 192.168.1.106@o2ib10 failed: rc = -19 Jun 7 18:47:07 soak-11 kernel: Lustre: soaked-OST0017-osc-MDT0003: Connection to soaked-OST0017 (at 192.168.1.106@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:47:07 soak-11 kernel: Lustre: Skipped 1 previous similar message Jun 7 18:47:07 soak-11 kernel: LustreError: Skipped 9 previous similar messages Jun 7 18:47:31 soak-11 kernel: Lustre: 4819:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1496861200/real 1496861200] req@ffff88074642b300 x1569566650052784/t0(0) o5->soaked-OST0017-osc-MDT0003@192.168.1.106@o2ib10:28/4 lens 432/432 e 0 to 1 dl 1496861251 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 7 18:47:31 soak-11 kernel: Lustre: 4819:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jun 7 18:47:31 soak-11 kernel: LustreError: 4819:0:(osp_precreate.c:872:osp_precreate_cleanup_orphans()) soaked-OST0017-osc-MDT0003: cannot cleanup orphans: rc = -11 Jun 7 18:48:46 soak-11 kernel: LNet: 4665:0:(o2iblnd_cb.c:3207:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib10: 4 seconds Jun 7 18:49:38 soak-11 kernel: Lustre: soaked-MDT0003: Connection restored to 192.168.1.107@o2ib10 (at 192.168.1.107@o2ib10) Jun 7 18:49:38 soak-11 kernel: Lustre: Skipped 5 previous similar messages Jun 7 18:49:42 soak-11 kernel: LustreError: 11-0: soaked-OST0005-osc-MDT0003: operation ost_statfs to node 192.168.1.106@o2ib10 failed: rc = -107 Jun 7 18:49:42 soak-11 kernel: Lustre: soaked-OST0005-osc-MDT0003: Connection to soaked-OST0005 (at 192.168.1.106@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:50:00 soak-11 kernel: LustreError: 11-0: soaked-OST000b-osc-MDT0003: operation ost_statfs to node 192.168.1.106@o2ib10 failed: rc = -107 Jun 7 18:50:00 soak-11 kernel: Lustre: soaked-OST000b-osc-MDT0003: Connection to soaked-OST000b (at 192.168.1.106@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:50:29 soak-11 kernel: LustreError: 11-0: soaked-OST0011-osc-MDT0003: operation ost_statfs to node 192.168.1.106@o2ib10 failed: rc = -107 Jun 7 18:50:29 soak-11 kernel: Lustre: soaked-OST0011-osc-MDT0003: Connection to soaked-OST0011 (at 192.168.1.106@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Jun 7 18:50:38 soak-11 kernel: INFO: task mdt01_003:4851 blocked for more than 120 seconds. Jun 7 18:50:38 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:50:38 soak-11 kernel: mdt01_003 D ffff880824b83250 0 4851 2 0x00000080 Jun 7 18:50:38 soak-11 kernel: ffff8803e7963568 0000000000000046 ffff880410dc6dd0 ffff8803e7963fd8 Jun 7 18:50:38 soak-11 kernel: ffff8803e7963fd8 ffff8803e7963fd8 ffff880410dc6dd0 ffff880410dc6dd0 Jun 7 18:50:38 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:50:38 soak-11 kernel: Call Trace: Jun 7 18:50:38 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:50:38 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:50:38 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:50:38 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:50:38 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:50:38 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:50:38 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:50:38 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:50:38 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:50:38 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:50:38 soak-11 kernel: INFO: task mdt00_031:5224 blocked for more than 120 seconds. Jun 7 18:50:38 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:50:39 soak-11 kernel: mdt00_031 D ffff880824b83250 0 5224 2 0x00000080 Jun 7 18:50:39 soak-11 kernel: ffff8807f46ef568 0000000000000046 ffff8808289faf10 ffff8807f46effd8 Jun 7 18:50:39 soak-11 kernel: ffff8807f46effd8 ffff8807f46effd8 ffff8808289faf10 ffff8808289faf10 Jun 7 18:50:39 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:50:39 soak-11 kernel: Call Trace: Jun 7 18:50:39 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:50:39 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:50:39 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:50:39 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:50:39 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:50:39 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:50:39 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:50:39 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:50:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:50:39 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:50:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:50:45 soak-11 kernel: Lustre: soaked-OST0011-osc-MDT0003: Connection restored to 192.168.1.107@o2ib10 (at 192.168.1.107@o2ib10) Jun 7 18:50:45 soak-11 kernel: Lustre: Skipped 6 previous similar messages Jun 7 18:51:38 soak-11 kernel: LNet: Service thread pid 5292 was inactive for 200.50s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 7 18:51:38 soak-11 kernel: Pid: 5292, comm: mdt00_050 Jun 7 18:51:38 soak-11 kernel: #012Call Trace: Jun 7 18:51:38 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:51:38 soak-11 kernel: [<ffffffff81689d64>] schedule_timeout+0x174/0x2c0 Jun 7 18:51:38 soak-11 kernel: [<ffffffff81096d60>] ? process_timeout+0x0/0x10 Jun 7 18:51:38 soak-11 kernel: [<ffffffffa0bafec1>] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Jun 7 18:51:38 soak-11 kernel: [<ffffffffa14e9f28>] osp_precreate_reserve+0x348/0x8e0 [osp] Jun 7 18:51:38 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:39 soak-11 kernel: [<ffffffffa14df3a3>] osp_declare_create+0x193/0x590 [osp] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0cd4599>] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa143aa4c>] lod_sub_declare_create+0xdc/0x210 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa1433ece>] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa14363c0>] lod_alloc_rr.constprop.18+0x700/0x1000 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa1439979>] lod_qos_prep_create+0x12b9/0x17f0 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa1240016>] ? osd_scrub_setup+0xb66/0x15e0 [osd_ldiskfs] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:51:39 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc] Jun 7 18:51:39 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:51:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:39 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:51:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:39 soak-11 kernel: Jun 7 18:51:39 soak-11 kernel: LustreError: dumping log to /tmp/lustre-log.1496861499.5292 Jun 7 18:51:41 soak-11 kernel: LNet: Service thread pid 5187 was inactive for 202.69s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 7 18:51:41 soak-11 kernel: Pid: 5187, comm: mdt01_028 Jun 7 18:51:41 soak-11 kernel: #012Call Trace: Jun 7 18:51:41 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:51:41 soak-11 kernel: [<ffffffff81689d64>] schedule_timeout+0x174/0x2c0 Jun 7 18:51:41 soak-11 kernel: [<ffffffff81096d60>] ? process_timeout+0x0/0x10 Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0bafec1>] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa14e9f28>] osp_precreate_reserve+0x348/0x8e0 [osp] Jun 7 18:51:41 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:41 soak-11 kernel: [<ffffffffa14df3a3>] osp_declare_create+0x193/0x590 [osp] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0cd4599>] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa143aa4c>] lod_sub_declare_create+0xdc/0x210 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa1433ece>] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa14363c0>] lod_alloc_rr.constprop.18+0x700/0x1000 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffff8131fc99>] ? __rwsem_do_wake+0x129/0x160 Jun 7 18:51:41 soak-11 kernel: [<ffffffffa1439979>] lod_qos_prep_create+0x12b9/0x17f0 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:51:41 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc] Jun 7 18:51:41 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:51:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:41 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:51:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:41 soak-11 kernel: Jun 7 18:51:50 soak-11 kernel: LNet: Service thread pid 5224 was inactive for 200.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 7 18:51:50 soak-11 kernel: Pid: 5224, comm: mdt00_031 Jun 7 18:51:50 soak-11 kernel: #012Call Trace: Jun 7 18:51:50 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:51:50 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:51:50 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:51:50 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:51:50 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:51:50 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:51:50 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:51:50 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:51:51 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc] Jun 7 18:51:51 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:51:51 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:51 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:51:51 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:51 soak-11 kernel: Jun 7 18:51:51 soak-11 kernel: LustreError: dumping log to /tmp/lustre-log.1496861511.5224 Jun 7 18:51:52 soak-11 kernel: LNet: Service thread pid 5292 completed after 214.21s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 7 18:51:57 soak-11 kernel: LNet: Service thread pid 5296 was inactive for 200.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 7 18:51:57 soak-11 kernel: Pid: 5296, comm: mdt01_046 Jun 7 18:51:57 soak-11 kernel: #012Call Trace: Jun 7 18:51:57 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:51:57 soak-11 kernel: [<ffffffff81689d64>] schedule_timeout+0x174/0x2c0 Jun 7 18:51:57 soak-11 kernel: [<ffffffff81096d60>] ? process_timeout+0x0/0x10 Jun 7 18:51:57 soak-11 kernel: [<ffffffffa0eeb300>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa0eebae1>] ldlm_completion_ast+0x5a1/0x910 [ptlrpc] Jun 7 18:51:57 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:57 soak-11 kernel: [<ffffffffa0eecbf0>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa0eeb540>] ? ldlm_completion_ast+0x0/0x910 [ptlrpc] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa1333360>] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa13435aa>] mdt_object_local_lock+0x50a/0xaf0 [mdt] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa1333360>] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Jun 7 18:51:57 soak-11 kernel: [<ffffffffa0eeb540>] ? ldlm_completion_ast+0x0/0x910 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f1e7b0>] ? lustre_msg_buf+0x0/0x60 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1343bee>] mdt_object_lock_internal+0x5e/0x300 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1344994>] mdt_getattr_name_lock+0x8a4/0x1900 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f1ea7c>] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1345ca0>] mdt_intent_getattr+0x2b0/0x480 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:58 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:58 soak-11 kernel: Jun 7 18:51:58 soak-11 kernel: LustreError: dumping log to /tmp/lustre-log.1496861518.5296 Jun 7 18:51:58 soak-11 kernel: Pid: 5267, comm: mdt00_035 Jun 7 18:51:58 soak-11 kernel: #012Call Trace: Jun 7 18:51:58 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:51:58 soak-11 kernel: [<ffffffff81689d64>] schedule_timeout+0x174/0x2c0 Jun 7 18:51:58 soak-11 kernel: [<ffffffff81096d60>] ? process_timeout+0x0/0x10 Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0eeb300>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0eebae1>] ldlm_completion_ast+0x5a1/0x910 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0eecbf0>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0eeb540>] ? ldlm_completion_ast+0x0/0x910 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1333360>] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa13435aa>] mdt_object_local_lock+0x50a/0xaf0 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1333360>] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0eeb540>] ? ldlm_completion_ast+0x0/0x910 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f1e7b0>] ? lustre_msg_buf+0x0/0x60 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1343bee>] mdt_object_lock_internal+0x5e/0x300 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1344994>] mdt_getattr_name_lock+0x8a4/0x1900 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f1ea7c>] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa1345ca0>] mdt_intent_getattr+0x2b0/0x480 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:51:58 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc] Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:58 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:51:58 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:51:58 soak-11 kernel: Jun 7 18:51:58 soak-11 kernel: LNet: Service thread pid 5186 was inactive for 201.17s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jun 7 18:52:39 soak-11 kernel: INFO: task mdt01_003:4851 blocked for more than 120 seconds. Jun 7 18:52:39 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:52:39 soak-11 kernel: mdt01_003 D ffff880824b83250 0 4851 2 0x00000080 Jun 7 18:52:39 soak-11 kernel: ffff8803e7963568 0000000000000046 ffff880410dc6dd0 ffff8803e7963fd8 Jun 7 18:52:39 soak-11 kernel: ffff8803e7963fd8 ffff8803e7963fd8 ffff880410dc6dd0 ffff880410dc6dd0 Jun 7 18:52:39 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:52:39 soak-11 kernel: Call Trace: Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:52:39 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:52:39 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:52:39 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:52:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:39 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:52:39 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:39 soak-11 kernel: INFO: task mdt01_029:5188 blocked for more than 120 seconds. Jun 7 18:52:39 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:52:39 soak-11 kernel: mdt01_029 D ffff880824b83250 0 5188 2 0x00000080 Jun 7 18:52:39 soak-11 kernel: ffff8807f822f568 0000000000000046 ffff880415c81f60 ffff8807f822ffd8 Jun 7 18:52:39 soak-11 kernel: ffff8807f822ffd8 ffff8807f822ffd8 ffff880415c81f60 ffff880415c81f60 Jun 7 18:52:39 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:52:39 soak-11 kernel: Call Trace: Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:52:39 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:52:39 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:52:39 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:52:39 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: INFO: task mdt00_031:5224 blocked for more than 120 seconds. Jun 7 18:52:40 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:52:40 soak-11 kernel: mdt00_031 D ffff880824b83250 0 5224 2 0x00000080 Jun 7 18:52:40 soak-11 kernel: ffff8807f46ef568 0000000000000046 ffff8808289faf10 ffff8807f46effd8 Jun 7 18:52:40 soak-11 kernel: ffff8807f46effd8 ffff8807f46effd8 ffff8808289faf10 ffff8808289faf10 Jun 7 18:52:40 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:52:40 soak-11 kernel: Call Trace: Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:52:40 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: INFO: task mdt00_032:5225 blocked for more than 120 seconds. Jun 7 18:52:40 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:52:40 soak-11 kernel: mdt00_032 D ffff880824b83250 0 5225 2 0x00000080 Jun 7 18:52:40 soak-11 kernel: ffff8807f407b568 0000000000000046 ffff8808289fedd0 ffff8807f407bfd8 Jun 7 18:52:40 soak-11 kernel: ffff8807f407bfd8 ffff8807f407bfd8 ffff8808289fedd0 ffff8808289fedd0 Jun 7 18:52:40 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:52:40 soak-11 kernel: Call Trace: Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:52:40 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:52:40 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:40 soak-11 kernel: INFO: task mdt00_054:5301 blocked for more than 120 seconds. Jun 7 18:52:40 soak-11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 7 18:52:40 soak-11 kernel: mdt00_054 D ffff880824b83250 0 5301 2 0x00000080 Jun 7 18:52:40 soak-11 kernel: ffff8807dc85b568 0000000000000046 ffff880821726dd0 ffff8807dc85bfd8 Jun 7 18:52:40 soak-11 kernel: ffff8807dc85bfd8 ffff8807dc85bfd8 ffff880821726dd0 ffff880821726dd0 Jun 7 18:52:40 soak-11 kernel: ffff880824b83240 ffff880824b83248 ffffffff00000000 ffff880824b83250 Jun 7 18:52:40 soak-11 kernel: Call Trace: Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168c3c9>] schedule+0x29/0x70 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168db75>] rwsem_down_write_failed+0x115/0x220 Jun 7 18:52:40 soak-11 kernel: [<ffffffff813275a7>] call_rwsem_down_write_failed+0x17/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffff8168b58d>] down_write+0x2d/0x30 Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1439164>] lod_qos_prep_create+0xaa4/0x17f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa11033c0>] ? qsd_op_begin+0xb0/0x4d0 [lquota] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa1243870>] ? osd_declare_qid+0x1f0/0x490 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa143a428>] lod_prepare_create+0x298/0x3f0 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa120723e>] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa142ef8e>] lod_declare_striped_create+0x1ee/0x970 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14314bc>] lod_declare_create+0x1fc/0x570 [lod] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa149c97f>] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa148e2f3>] mdd_declare_create+0x53/0xe20 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa14920d9>] mdd_create+0x7c9/0x1330 [mdd] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa13683a2>] mdt_reint_open+0x2192/0x31a0 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa0d09df9>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa134dc13>] ? ucred_set_jobid+0x53/0x70 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa135d530>] mdt_reint_rec+0x80/0x210 [mdt] Jun 7 18:52:40 soak-11 kernel: [<ffffffffa133f49b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa133f9c2>] mdt_intent_reint+0x162/0x430 [mdt] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa134a73e>] mdt_intent_policy+0x43e/0xc70 [mdt] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0ed7e3f>] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0ed1167>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0efaab3>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f22cb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f80012>] tgt_enqueue+0x62/0x210 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f832d5>] tgt_request_handle+0x915/0x1360 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f2c493>] ptlrpc_server_handle_request+0x233/0xa90 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f29c88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffff810c54d2>] ? default_wake_function+0x12/0x20 Jun 7 18:52:41 soak-11 kernel: [<ffffffff810ba628>] ? __wake_up_common+0x58/0x90 Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f304a0>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0 Jun 7 18:52:41 soak-11 kernel: [<ffffffffa0f2fa00>] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] Jun 7 18:52:41 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:52:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:52:41 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:52:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 Jun 7 18:53:32 soak-11 kernel: LNet: Service thread pid 5187 completed after 314.21s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 7 18:53:32 soak-11 kernel: LNet: Skipped 8 previous similar messages Jun 7 18:53:35 soak-11 kernel: LNet: Service thread pid 5224 completed after 304.87s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 7 18:53:41 soak-11 kernel: LustreError: 4653:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) ASSERTION( lock->l_granted_mode & (LCK_PW | LCK_EX) ) failed: Jun 7 18:53:41 soak-11 kernel: LustreError: 4653:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) LBUG Jun 7 18:53:41 soak-11 kernel: Pid: 4653, comm: ptlrpc_hr01_005 Jun 7 18:53:41 soak-11 kernel: #012Call Trace: Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0baf7ee>] libcfs_call_trace+0x4e/0x60 [libcfs] Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0baf87c>] lbug_with_loc+0x4c/0xb0 [libcfs] Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0ed283b>] ldlm_lock_downgrade+0x19b/0x1d0 [ptlrpc] Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0f297f8>] ptlrpc_handle_rs+0x5c8/0x700 [ptlrpc] Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0f29a15>] ptlrpc_hr_main+0xe5/0x2c0 [ptlrpc] Jun 7 18:53:41 soak-11 kernel: [<ffffffff810c54c0>] ? default_wake_function+0x0/0x20 Jun 7 18:53:41 soak-11 kernel: [<ffffffffa0f29930>] ? ptlrpc_hr_main+0x0/0x2c0 [ptlrpc] Jun 7 18:53:41 soak-11 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0 Jun 7 18:53:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:53:41 soak-11 kernel: [<ffffffff81697318>] ret_from_fork+0x58/0x90 Jun 7 18:53:41 soak-11 kernel: [<ffffffff810b0980>] ? kthread+0x0/0xe0 Jun 7 18:53:41 soak-11 kernel: Jun 7 18:53:41 soak-11 kernel: Kernel panic - not syncing: LBUG Jun 7 18:53:41 soak-11 kernel: CPU: 13 PID: 4653 Comm: ptlrpc_hr01_005 Tainted: P OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1