Feb 21 04:13:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42b07aa2-0e4f-a664-521f-8af46f8ac255 (at 10.210.47.121@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f9b809400, cur 1519215185 expire 1519215035 last 1519214958 Feb 21 04:26:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42b07aa2-0e4f-a664-521f-8af46f8ac255 (at 10.210.47.121@o2ib3) Feb 21 06:01:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ee345a0-1ac4-c87a-a275-576c70390233 (at 10.8.28.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d86295800, cur 1519221660 expire 1519221510 last 1519221433 Feb 21 07:37:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to edc9ac7c-8da6-16ad-728b-6d71addd8cf0 (at 10.8.29.4@o2ib6) Feb 21 07:37:34 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226853/real 1519226853] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519227454 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:37:34 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 07:37:34 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:37:34 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:37:34 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Feb 21 07:37:35 oak-md1-s2 kernel: Lustre: 102693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226853/real 1519226853] req@ffff880e4de08c00 x1592931637385856/t0(0) o6->oak-OST0031-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519227454 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:37:35 oak-md1-s2 kernel: Lustre: oak-OST0031-osc-MDT0000: Connection to oak-OST0031 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:37:55 oak-md1-s2 kernel: Lustre: 102711:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226874/real 1519226874] req@ffff881f60b78900 x1592931637391712/t0(0) o6->oak-OST003b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519227475 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:37:55 oak-md1-s2 kernel: Lustre: oak-OST003b-osc-MDT0000: Connection to oak-OST003b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:37:55 oak-md1-s2 kernel: Lustre: oak-OST003b-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:37:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:38:17 oak-md1-s2 kernel: Lustre: 102714:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226895/real 1519226895] req@ffff881f7c765700 x1592931637398080/t0(0) o6->oak-OST0035-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 12 to 1 dl 1519227496 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:38:17 oak-md1-s2 kernel: Lustre: oak-OST0035-osc-MDT0000: Connection to oak-OST0035 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:38:17 oak-md1-s2 kernel: Lustre: oak-OST0035-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:38:37 oak-md1-s2 kernel: Lustre: 102698:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226916/real 1519226916] req@ffff880311533900 x1592931637403904/t0(0) o6->oak-OST0053-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 8 to 1 dl 1519227517 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:38:37 oak-md1-s2 kernel: Lustre: oak-OST0053-osc-MDT0000: Connection to oak-OST0053 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:38:37 oak-md1-s2 kernel: Lustre: oak-OST0053-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:38:52 oak-md1-s2 kernel: Lustre: 102698:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226930/real 1519226930] req@ffff88088dff5a00 x1592931637408416/t0(0) o6->oak-OST003d-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 7 to 1 dl 1519227531 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:38:52 oak-md1-s2 kernel: Lustre: oak-OST003d-osc-MDT0000: Connection to oak-OST003d (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:38:52 oak-md1-s2 kernel: Lustre: oak-OST003d-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:39:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97798af8-eef3-dbd4-8e85-fef451a162f3 (at 10.210.44.48@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f57676800, cur 1519227556 expire 1519227406 last 1519227329 Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: 102704:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519226969/real 1519226969] req@ffff88093ede3f00 x1592931637419280/t0(0) o6->oak-OST004f-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 5 to 1 dl 1519227570 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: 102704:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: oak-OST004f-osc-MDT0000: Connection to oak-OST004f (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: oak-OST004f-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:39:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:39:30 oak-md1-s2 kernel: LustreError: 102964:0:(osp_precreate.c:619:osp_precreate_send()) oak-OST0031-osc-MDT0000: can't precreate: rc = -107 Feb 21 07:40:02 oak-md1-s2 kernel: Lustre: 102712:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519227000/real 1519227000] req@ffff881f12aa9e00 x1592931637428352/t0(0) o6->oak-OST0045-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 4 to 1 dl 1519227601 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:40:02 oak-md1-s2 kernel: Lustre: 102712:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 07:40:02 oak-md1-s2 kernel: Lustre: oak-OST0045-osc-MDT0000: Connection to oak-OST0045 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:40:02 oak-md1-s2 kernel: Lustre: oak-OST0045-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:41:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 75d30f74-7923-738b-70bc-bc5c47963847 (at 10.8.28.1@o2ib6) Feb 21 07:41:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:41:15 oak-md1-s2 kernel: Lustre: 102694:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519227074/real 1519227074] req@ffff8804a5adb600 x1592931637449072/t0(0) o6->oak-OST004b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 3 to 1 dl 1519227675 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:41:15 oak-md1-s2 kernel: Lustre: 102694:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 07:41:15 oak-md1-s2 kernel: Lustre: oak-OST004b-osc-MDT0000: Connection to oak-OST004b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:41:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: 102702:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519227206/real 1519227206] req@ffff8809d7761200 x1592931637486896/t0(0) o6->oak-OST0033-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519227807 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: 102702:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: oak-OST0033-osc-MDT0000: Connection to oak-OST0033 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: oak-OST0033-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:43:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Feb 21 07:45:33 oak-md1-s2 kernel: INFO: task mdt00_015:103280 blocked for more than 120 seconds. Feb 21 07:45:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:45:33 oak-md1-s2 kernel: mdt00_015 D ffffffff00000000 0 103280 2 0x00000080 Feb 21 07:45:33 oak-md1-s2 kernel: ffff88026bf73558 0000000000000046 ffff881e5f5b0000 ffff88026bf73fd8 Feb 21 07:45:33 oak-md1-s2 kernel: ffff88026bf73fd8 ffff88026bf73fd8 ffff881e5f5b0000 ffff881e5f5b0000 Feb 21 07:45:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:45:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:45:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:45:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:45:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:45:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:45:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 07:45:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:45:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:45:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:45:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:45:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:45:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:45:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:45:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:45:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: INFO: task mdt00_000:102832 blocked for more than 120 seconds. Feb 21 07:47:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:47:33 oak-md1-s2 kernel: mdt00_000 D ffffffff00000000 0 102832 2 0x00000080 Feb 21 07:47:33 oak-md1-s2 kernel: ffff880fff05f558 0000000000000046 ffff880cb103dee0 ffff880fff05ffd8 Feb 21 07:47:33 oak-md1-s2 kernel: ffff880fff05ffd8 ffff880fff05ffd8 ffff880cb103dee0 ffff880cb103dee0 Feb 21 07:47:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:47:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:47:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:47:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:47:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:47:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: INFO: task mdt00_007:103156 blocked for more than 120 seconds. Feb 21 07:47:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:47:33 oak-md1-s2 kernel: mdt00_007 D ffffffff00000000 0 103156 2 0x00000080 Feb 21 07:47:33 oak-md1-s2 kernel: ffff881001d9f478 0000000000000046 ffff88102da61fa0 ffff881001d9ffd8 Feb 21 07:47:33 oak-md1-s2 kernel: ffff881001d9ffd8 ffff881001d9ffd8 ffff88102da61fa0 ffff88102da61fa0 Feb 21 07:47:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:47:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:47:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:47:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:47:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: INFO: task mdt00_010:103184 blocked for more than 120 seconds. Feb 21 07:47:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:47:33 oak-md1-s2 kernel: mdt00_010 D ffffffff00000000 0 103184 2 0x00000080 Feb 21 07:47:33 oak-md1-s2 kernel: ffff88102c75b478 0000000000000046 ffff881e5d9a4f10 ffff88102c75bfd8 Feb 21 07:47:33 oak-md1-s2 kernel: ffff88102c75bfd8 ffff88102c75bfd8 ffff881e5d9a4f10 ffff881e5d9a4f10 Feb 21 07:47:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:47:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:47:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:47:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:47:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? __getblk+0x2d/0x300 Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:47:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:47:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: 102714:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519227475/real 1519227475] req@ffff881f60b7ce00 x1592931637469040/t0(0) o6->oak-OST003b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519228076 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: 102714:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: oak-OST003b-osc-MDT0000: Connection to oak-OST003b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: oak-OST003b-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:47:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Feb 21 07:50:10 oak-md1-s2 kernel: LustreError: 103000:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0043-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:52:06 oak-md1-s2 kernel: LustreError: 102964:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0031-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:52:38 oak-md1-s2 kernel: LustreError: 103004:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0045-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:53:51 oak-md1-s2 kernel: LustreError: 103016:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST004b-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:55:19 oak-md1-s2 kernel: LustreError: 103028:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0051-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:55:33 oak-md1-s2 kernel: INFO: task mdt00_012:103187 blocked for more than 120 seconds. Feb 21 07:55:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:55:33 oak-md1-s2 kernel: mdt00_012 D ffffffff00000000 0 103187 2 0x00000080 Feb 21 07:55:33 oak-md1-s2 kernel: ffff88103da7f478 0000000000000046 ffff88103e0f8000 ffff88103da7ffd8 Feb 21 07:55:33 oak-md1-s2 kernel: ffff88103da7ffd8 ffff88103da7ffd8 ffff88103e0f8000 ffff88103e0f8000 Feb 21 07:55:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:55:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:55:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:55:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:55:33 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 07:55:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:55:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:55:33 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? osd_ldiskfs_write_record+0x2d2/0x410 [osd_ldiskfs] Feb 21 07:55:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 07:55:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:55:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:55:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:55:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:55:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:55:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:55:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:55:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:55:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:56:03 oak-md1-s2 kernel: LustreError: 102968:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0033-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 07:57:33 oak-md1-s2 kernel: INFO: task mdt00_013:103189 blocked for more than 120 seconds. Feb 21 07:57:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 07:57:33 oak-md1-s2 kernel: mdt00_013 D ffffffff00000000 0 103189 2 0x00000080 Feb 21 07:57:33 oak-md1-s2 kernel: ffff881035603558 0000000000000046 ffff880ffd065ee0 ffff881035603fd8 Feb 21 07:57:33 oak-md1-s2 kernel: ffff881035603fd8 ffff881035603fd8 ffff880ffd065ee0 ffff880ffd065ee0 Feb 21 07:57:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 07:57:33 oak-md1-s2 kernel: Call Trace: Feb 21 07:57:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 07:57:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 07:57:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 07:57:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 07:57:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 07:57:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 07:57:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 07:57:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 07:57:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 07:57:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 07:57:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 07:57:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:57:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 07:57:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519228055/real 1519228055] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519228656 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 07:57:36 oak-md1-s2 kernel: Lustre: Skipped 18 previous similar messages Feb 21 08:01:33 oak-md1-s2 kernel: INFO: task mdt00_004:103144 blocked for more than 120 seconds. Feb 21 08:01:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 08:01:33 oak-md1-s2 kernel: mdt00_004 D ffffffff00000000 0 103144 2 0x00000080 Feb 21 08:01:33 oak-md1-s2 kernel: ffff881038b1f478 0000000000000046 ffff88103e0fdee0 ffff881038b1ffd8 Feb 21 08:01:33 oak-md1-s2 kernel: ffff881038b1ffd8 ffff881038b1ffd8 ffff88103e0fdee0 ffff88103e0fdee0 Feb 21 08:01:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 08:01:33 oak-md1-s2 kernel: Call Trace: Feb 21 08:01:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 08:01:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 08:01:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 08:01:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 08:01:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 08:01:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 08:01:33 oak-md1-s2 kernel: INFO: task mdt00_018:177314 blocked for more than 120 seconds. Feb 21 08:01:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 08:01:33 oak-md1-s2 kernel: mdt00_018 D ffffffff00000000 0 177314 2 0x00000080 Feb 21 08:01:33 oak-md1-s2 kernel: ffff881deb02b558 0000000000000046 ffff881e5f5b0fd0 ffff881deb02bfd8 Feb 21 08:01:33 oak-md1-s2 kernel: ffff881deb02bfd8 ffff881deb02bfd8 ffff881e5f5b0fd0 ffff881e5f5b0fd0 Feb 21 08:01:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 08:01:33 oak-md1-s2 kernel: Call Trace: Feb 21 08:01:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 08:01:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 08:01:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 08:01:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 08:01:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 08:01:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 08:01:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 08:01:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 08:01:35 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:619:osp_precreate_send()) oak-OST0037-osc-MDT0000: can't precreate: rc = -107 Feb 21 08:02:47 oak-md1-s2 kernel: LustreError: 103000:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0043-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:04:43 oak-md1-s2 kernel: LustreError: 102964:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0031-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:06:28 oak-md1-s2 kernel: LustreError: 103016:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST004b-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:06:28 oak-md1-s2 kernel: LustreError: 103016:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message Feb 21 08:06:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26daf71a-6b1d-9025-91c4-b5a509a534c6 (at 10.210.47.108@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fa63c2c00, cur 1519229207 expire 1519229057 last 1519228980 Feb 21 08:06:47 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519228656/real 1519228656] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519229257 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:07:37 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 08:08:40 oak-md1-s2 kernel: LustreError: 102968:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0033-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:08:40 oak-md1-s2 kernel: LustreError: 102968:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message Feb 21 08:14:11 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519229257/real 1519229257] req@ffff881fdc4a5700 x1592931637413312/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 6 to 1 dl 1519229858 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:17:38 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 08:26:48 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:26:48 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519229858/real 1519229858] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519230459 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:27:39 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:31:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5644f3fd-9d24-3aa5-47fa-f350aad949ce (at 10.9.113.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d88e9c400, cur 1519230700 expire 1519230550 last 1519230473 Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519230459/real 1519230459] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519231060 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:37:40 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:39:25 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:39:25 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519231060/real 1519231060] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519231661 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:47:41 oak-md1-s2 kernel: Lustre: Skipped 20 previous similar messages Feb 21 08:52:02 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 08:52:02 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519231661/real 1519231661] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519232262 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 08:57:42 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Feb 21 09:00:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1b95f2b6-37a3-daab-4a57-c536e4b4a9e1 (at 10.9.112.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103dbf2c00, cur 1519232413 expire 1519232263 last 1519232186 Feb 21 09:04:39 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 09:04:39 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519232262/real 1519232262] req@ffff881fdc4a5700 x1592931637413312/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 6 to 1 dl 1519232863 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:07:43 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 09:17:16 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 09:17:16 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519232863/real 1519232863] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519233464 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:17:44 oak-md1-s2 kernel: Lustre: Skipped 20 previous similar messages Feb 21 09:21:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51a772ba-1402-58f0-7b77-ccfe83dcdfe6 (at 10.9.101.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fcb6fa800, cur 1519233716 expire 1519233566 last 1519233489 Feb 21 09:21:56 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519233464/real 1519233464] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519234065 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:27:45 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 09:29:53 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 09:29:53 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519234065/real 1519234065] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519234666 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:37:46 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 09:42:30 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 09:42:30 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519234666/real 1519234666] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519235267 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Feb 21 09:47:47 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Feb 21 09:48:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf27dbf4-35f5-5de8-77cb-bd6a58a12004 (at 10.210.47.108@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202d736400, cur 1519235326 expire 1519235176 last 1519235099 Feb 21 09:55:07 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 09:55:07 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519235267/real 1519235267] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519235868 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 09:57:48 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 10:07:44 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 10:07:44 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519235868/real 1519235868] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519236469 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:07:49 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Feb 21 10:17:49 oak-md1-s2 kernel: LustreError: 102992:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST003f-osc-MDT0000: cannot cleanup orphans: rc = -11 Feb 21 10:17:49 oak-md1-s2 kernel: LustreError: 102992:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519236469/real 1519236469] req@ffff881e5ce59200 x1592931637385888/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 23 to 1 dl 1519237070 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:17:50 oak-md1-s2 kernel: Lustre: 102709:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519237070/real 1519237070] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519237671 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:27:51 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 10:27:56 oak-md1-s2 kernel: LustreError: 103024:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST004f-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 10:27:56 oak-md1-s2 kernel: LustreError: 103024:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 12 previous similar messages Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519237671/real 1519237671] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519238272 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:37:52 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:38:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a210634a-ad6d-c489-dc14-1a4069fdc746 (at 10.9.0.62@o2ib4) reconnecting Feb 21 10:38:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2d21ac44-ee87-ac13-a569-7b484f84fdeb (at 10.9.112.4@o2ib4) reconnecting Feb 21 10:38:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a8ab1186-4f11-0c52-3ba4-bf0295ae3f90 (at 10.9.112.1@o2ib4) reconnecting Feb 21 10:38:44 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:38:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c2b61871-ebd9-8108-2c0d-d32daffdb860 (at 10.9.113.8@o2ib4) reconnecting Feb 21 10:38:46 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Feb 21 10:38:58 oak-md1-s2 kernel: LustreError: 102984:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST003b-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 10:38:58 oak-md1-s2 kernel: LustreError: 102984:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages Feb 21 10:39:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8089da30-2e6e-f45e-8f52-4de3332ae98a (at 10.9.105.28@o2ib4) reconnecting Feb 21 10:39:30 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519238272/real 1519238272] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519238873 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:47:53 oak-md1-s2 kernel: Lustre: Skipped 56 previous similar messages Feb 21 10:49:03 oak-md1-s2 kernel: LustreError: 103012:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0049-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 10:49:03 oak-md1-s2 kernel: LustreError: 103012:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 11 previous similar messages Feb 21 10:55:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d41b8d5a-f1ae-3f2a-fc91-7fd6f565b4d5 (at 10.9.112.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f1dc69000, cur 1519239354 expire 1519239204 last 1519239127 Feb 21 10:55:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519238873/real 1519238873] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519239474 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 10:57:54 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:59:25 oak-md1-s2 kernel: LustreError: 103000:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0043-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 10:59:25 oak-md1-s2 kernel: LustreError: 103000:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 12 previous similar messages Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: 102707:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519239474/real 1519239474] req@ffff881fdc4a4800 x1592931637538768/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 2 to 1 dl 1519240075 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519239474/real 1519239474] req@ffff881fdc4a5700 x1592931637413312/t0(0) o6->oak-OST0043-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 6 to 1 dl 1519240075 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: 102710:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection to oak-OST0043 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: oak-OST0043-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Feb 21 11:07:55 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 11:10:49 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST0037-osc-MDT0000: cannot cleanup orphans: rc = -107 Feb 21 11:10:49 oak-md1-s2 kernel: LustreError: 102976:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 13 previous similar messages Feb 21 11:11:18 oak-md1-s2 kernel: LNetError: 102679:0:(o2iblnd_cb.c:3147:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds Feb 21 11:11:18 oak-md1-s2 kernel: LNetError: 102679:0:(o2iblnd_cb.c:3222:kiblnd_check_conns()) Timed out RDMA with 10.0.2.106@o2ib5 (62): c: 0, oc: 0, rc: 8 Feb 21 11:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 37aaf50c-8c60-8b4c-4dd5-8196b363bec1 (at 10.9.104.69@o2ib4) reconnecting Feb 21 11:11:24 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Feb 21 11:11:30 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 24 seconds Feb 21 11:11:43 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 37 seconds Feb 21 11:11:43 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 21 11:11:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f2920406-cd9c-53c3-e051-5ed536cbb14d (at 10.210.47.232@o2ib3) reconnecting Feb 21 11:11:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7f20b8b8-e898-e588-84ce-effd8b632314 (at 10.9.102.63@o2ib4) reconnecting Feb 21 11:11:54 oak-md1-s2 kernel: Lustre: Skipped 73 previous similar messages Feb 21 11:11:56 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 50 seconds Feb 21 11:11:56 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 21 11:11:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9dec5549-ecf4-2f96-f51a-8fdcea59727d (at 10.210.45.85@o2ib3) reconnecting Feb 21 11:11:58 oak-md1-s2 kernel: Lustre: Skipped 56 previous similar messages Feb 21 11:12:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 57f51183-ab6e-7eb3-a623-7bc2f5e9e278 (at 10.210.46.120@o2ib3) reconnecting Feb 21 11:12:06 oak-md1-s2 kernel: Lustre: Skipped 111 previous similar messages Feb 21 11:12:08 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 62 seconds Feb 21 11:12:08 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 21 11:12:21 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 75 seconds Feb 21 11:12:21 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 21 11:12:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a46df59f-b465-42d6-bc8b-20cf887223d0 (at 10.9.104.17@o2ib4) reconnecting Feb 21 11:12:24 oak-md1-s2 kernel: Lustre: Skipped 178 previous similar messages Feb 21 11:12:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a4212d39-5fb4-3ae1-cb08-f784222c4575 (at 10.9.102.52@o2ib4) reconnecting Feb 21 11:12:57 oak-md1-s2 kernel: Lustre: Skipped 46 previous similar messages Feb 21 11:12:59 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 1 seconds Feb 21 11:12:59 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 10 previous similar messages Feb 21 11:13:37 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 4 seconds Feb 21 11:13:37 oak-md1-s2 kernel: LNet: 102679:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 21 11:14:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.105@o2ib5, removing former export from 10.0.2.106@o2ib5 Feb 21 11:14:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.105@o2ib5, removing former export from 10.0.2.106@o2ib5 Feb 21 11:14:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 11:14:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.105@o2ib5, removing former export from 10.0.2.106@o2ib5 Feb 21 11:14:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 11:14:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 036ee53b-7e10-38a2-d8f4-83e498f1a9ac (at 10.210.44.250@o2ib3) reconnecting Feb 21 11:14:07 oak-md1-s2 kernel: Lustre: Skipped 445 previous similar messages Feb 21 11:14:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.105@o2ib5, removing former export from 10.0.2.106@o2ib5 Feb 21 11:14:08 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Feb 21 11:14:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.105@o2ib5, removing former export from 10.0.2.106@o2ib5 Feb 21 11:14:12 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Feb 21 11:14:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6b51dbf-e757-4f34-40af-1ce1e2526fd1 (at 10.9.101.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db9ad1000, cur 1519240478 expire 1519240328 last 1519240251 Feb 21 11:17:55 oak-md1-s2 kernel: Lustre: 102968:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519239919/real 1519239919] req@ffff881e168f7500 x1592931640572432/t0(0) o5->oak-OST0033-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 432/432 e 0 to 1 dl 1519240675 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Feb 21 11:17:55 oak-md1-s2 kernel: Lustre: 102968:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 178 previous similar messages Feb 21 11:20:54 oak-md1-s2 kernel: LustreError: 102992:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) oak-OST003f-osc-MDT0000: cannot cleanup orphans: rc = -11 Feb 21 11:20:54 oak-md1-s2 kernel: LustreError: 102992:0:(osp_precreate.c:903:osp_precreate_cleanup_orphans()) Skipped 21 previous similar messages Feb 21 11:28:39 oak-md1-s2 kernel: Lustre: oak-OST0051-osc-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) Feb 21 11:28:39 oak-md1-s2 kernel: Lustre: Skipped 946 previous similar messages Feb 21 11:28:47 oak-md1-s2 kernel: LustreError: 102692:0:(client.c:3007:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff88020ad62a00 x1592931638579664/t4295689592(4295689592) o6->oak-OST0053-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 664/400 e 5 to 0 dl 1519241348 ref 2 fl Interpret:R/4/0 rc -2/-2 Feb 21 11:31:33 oak-md1-s2 kernel: INFO: task mdt01_002:102837 blocked for more than 120 seconds. Feb 21 11:31:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 11:31:33 oak-md1-s2 kernel: mdt01_002 D ffffffff00000000 0 102837 2 0x00000080 Feb 21 11:31:33 oak-md1-s2 kernel: ffff88102428f558 0000000000000046 ffff88027975cf10 ffff88102428ffd8 Feb 21 11:31:33 oak-md1-s2 kernel: ffff88102428ffd8 ffff88102428ffd8 ffff88027975cf10 ffff88027975cf10 Feb 21 11:31:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 11:31:33 oak-md1-s2 kernel: Call Trace: Feb 21 11:31:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:31:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:31:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:31:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:31:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 11:31:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 11:31:33 oak-md1-s2 kernel: INFO: task mdt00_004:103144 blocked for more than 120 seconds. Feb 21 11:31:33 oak-md1-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 21 11:31:33 oak-md1-s2 kernel: mdt00_004 D ffffffff00000000 0 103144 2 0x00000080 Feb 21 11:31:33 oak-md1-s2 kernel: ffff881038b1f478 0000000000000046 ffff88103e0fdee0 ffff881038b1ffd8 Feb 21 11:31:33 oak-md1-s2 kernel: ffff881038b1ffd8 ffff881038b1ffd8 ffff88103e0fdee0 ffff88103e0fdee0 Feb 21 11:31:33 oak-md1-s2 kernel: ffff88036428f248 ffff88036428f240 fffffffe00000001 ffffffff00000000 Feb 21 11:31:33 oak-md1-s2 kernel: Call Trace: Feb 21 11:31:33 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:31:33 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 11:31:33 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:31:33 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:31:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Feb 21 11:31:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 11:31:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:31:33 oak-md1-s2 kernel: [] ? insert_kthread_work+0x40/0x40 Feb 21 11:32:05 oak-md1-s2 kernel: LNet: Service thread pid 103179 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 11:32:05 oak-md1-s2 kernel: Pid: 103179, comm: mdt00_009 Feb 21 11:32:05 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:32:05 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:32:05 oak-md1-s2 kernel: [] rwsem_down_read_failed+0x10d/0x1a0 Feb 21 11:32:05 oak-md1-s2 kernel: [] call_rwsem_down_read_failed+0x18/0x30 Feb 21 11:32:05 oak-md1-s2 kernel: [] down_read+0x20/0x40 Feb 21 11:32:05 oak-md1-s2 kernel: [] lod_alloc_rr.constprop.18+0x22c/0x1000 [lod] Feb 21 11:32:05 oak-md1-s2 kernel: [] lod_qos_prep_create+0x12b9/0x17f0 [lod] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:32:05 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:32:05 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:32:05 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:32:05 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:32:05 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:32:05 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:32:05 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:05 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:32:05 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:05 oak-md1-s2 kernel: Feb 21 11:32:05 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241525.103179 Feb 21 11:32:59 oak-md1-s2 kernel: LNet: Service thread pid 103280 was inactive for 200.25s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 11:32:59 oak-md1-s2 kernel: Pid: 103280, comm: mdt00_015 Feb 21 11:32:59 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:32:59 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:32:59 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:32:59 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:32:59 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: Feb 21 11:32:59 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241579.103280 Feb 21 11:32:59 oak-md1-s2 kernel: Pid: 103139, comm: mdt01_004 Feb 21 11:32:59 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:32:59 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:32:59 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:32:59 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:32:59 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: Feb 21 11:32:59 oak-md1-s2 kernel: Pid: 103188, comm: mdt01_013 Feb 21 11:32:59 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:32:59 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:32:59 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Feb 21 11:32:59 oak-md1-s2 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? update_curr+0x104/0x190 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Feb 21 11:32:59 oak-md1-s2 kernel: [] osp_declare_create+0x193/0x590 [osp] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_sub_declare_create+0xdc/0x210 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0xea2/0x1590 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? wake_up_q+0x5b/0x80 Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: Feb 21 11:32:59 oak-md1-s2 kernel: Pid: 102836, comm: mdt01_001 Feb 21 11:32:59 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:32:59 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:32:59 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:32:59 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:32:59 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:32:59 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:32:59 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:32:59 oak-md1-s2 kernel: Feb 21 11:32:59 oak-md1-s2 kernel: LNet: Service thread pid 103171 was inactive for 200.22s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:33:14 oak-md1-s2 kernel: LNet: Service thread pid 220674 was inactive for 200.57s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:33:14 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241594.220674 Feb 21 11:33:16 oak-md1-s2 kernel: LNet: Service thread pid 199679 was inactive for 200.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:33:16 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241596.199679 Feb 21 11:33:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6b51dbf-e757-4f34-40af-1ce1e2526fd1 (at 10.9.101.54@o2ib4) Feb 21 11:33:44 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Feb 21 11:33:44 oak-md1-s2 kernel: LNet: Service thread pid 103188 completed after 245.40s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:33:44 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Feb 21 11:33:57 oak-md1-s2 kernel: LNet: Service thread pid 102846 was inactive for 212.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:33:57 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241637.102846 Feb 21 11:34:29 oak-md1-s2 kernel: LNet: Service thread pid 220684 was inactive for 250.05s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:34:29 oak-md1-s2 kernel: LNet: Skipped 1 previous similar message Feb 21 11:34:29 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241669.220684 Feb 21 11:34:35 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241675.103156 Feb 21 11:34:36 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241676.103142 Feb 21 11:34:38 oak-md1-s2 kernel: LNet: Service thread pid 199942 was inactive for 250.10s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:34:38 oak-md1-s2 kernel: LNet: Skipped 7 previous similar messages Feb 21 11:34:38 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241678.199942 Feb 21 11:34:39 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241679.220682 Feb 21 11:34:52 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241692.148966 Feb 21 11:34:53 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241693.220676 Feb 21 11:35:24 oak-md1-s2 kernel: LNet: Service thread pid 103179 completed after 400.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:35:24 oak-md1-s2 kernel: LNet: Skipped 2 previous similar messages Feb 21 11:35:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ee1d0378-e033-452a-77a7-2c16b8413b0f (at 10.9.101.34@o2ib4) reconnecting Feb 21 11:35:24 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Feb 21 11:35:25 oak-md1-s2 kernel: LustreError: 103142:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241425, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8801fa7b4200/0xec47b06a6b3b9c78 lrc: 3/0,1 mode: --/CW res: [0x20000f271:0x2562:0x0].0x0 bits 0x2 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103142 timeout: 0 lvb_type: 0 Feb 21 11:35:25 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241725.103142 Feb 21 11:35:27 oak-md1-s2 kernel: LustreError: 102833:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241427, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880afe56c400/0xec47b06a6b3d396a lrc: 3/0,1 mode: --/CW res: [0x20000f271:0x2562:0x0].0x0 bits 0x2 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 102833 timeout: 0 lvb_type: 0 Feb 21 11:35:42 oak-md1-s2 kernel: LustreError: 220676:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241442, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880e0908aa00/0xec47b06a6b44312f lrc: 3/1,0 mode: --/PR res: [0x20000db01:0xda33:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 220676 timeout: 0 lvb_type: 0 Feb 21 11:37:04 oak-md1-s2 kernel: LNet: Service thread pid 102832 completed after 399.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:37:04 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Feb 21 11:37:06 oak-md1-s2 kernel: LustreError: 220680:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241526, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880248745a00/0xec47b06a6b61994d lrc: 3/1,0 mode: --/PR res: [0x20000c387:0x1420f:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 220680 timeout: 0 lvb_type: 0 Feb 21 11:38:44 oak-md1-s2 kernel: LNet: Service thread pid 103154 completed after 499.94s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:38:57 oak-md1-s2 kernel: LNet: Service thread pid 103147 was inactive for 412.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 11:38:57 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Feb 21 11:38:57 oak-md1-s2 kernel: Pid: 103147, comm: mdt01_006 Feb 21 11:38:57 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:38:57 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:38:57 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:38:57 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: Feb 21 11:38:57 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241937.103147 Feb 21 11:38:57 oak-md1-s2 kernel: Pid: 103186, comm: mdt01_012 Feb 21 11:38:57 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:38:57 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:38:57 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:38:57 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: Feb 21 11:38:57 oak-md1-s2 kernel: Pid: 103169, comm: mdt01_009 Feb 21 11:38:57 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:38:57 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:38:57 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:38:57 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? __getblk+0x2d/0x300 Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:38:57 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:38:57 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:38:57 oak-md1-s2 kernel: Feb 21 11:39:06 oak-md1-s2 kernel: LNet: Service thread pid 103148 was inactive for 412.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 11:39:06 oak-md1-s2 kernel: LNet: Skipped 2 previous similar messages Feb 21 11:39:06 oak-md1-s2 kernel: Pid: 103148, comm: mdt01_007 Feb 21 11:39:06 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:39:06 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:39:06 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:39:06 oak-md1-s2 kernel: [] ? osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs] Feb 21 11:39:06 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:39:06 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:39:06 oak-md1-s2 kernel: [] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? qsd_op_begin0+0x181/0x940 [lquota] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? __getblk+0x2d/0x300 Feb 21 11:39:06 oak-md1-s2 kernel: [] lod_qos_prep_create+0x1291/0x17f0 [lod] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 11:39:06 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:39:06 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:39:06 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x5e2/0xa30 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:39:06 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:39:06 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:39:06 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:39:06 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:39:06 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:39:06 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:39:06 oak-md1-s2 kernel: Feb 21 11:39:06 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241946.103148 Feb 21 11:39:17 oak-md1-s2 kernel: LustreError: 103189:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241657, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff88097a2aca00/0xec47b06a6b85b33e lrc: 3/1,0 mode: --/PR res: [0x20000c473:0x436:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103189 timeout: 0 lvb_type: 0 Feb 21 11:39:20 oak-md1-s2 kernel: LNet: Service thread pid 103172 was inactive for 412.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 11:39:20 oak-md1-s2 kernel: Pid: 103172, comm: mdt01_011 Feb 21 11:39:20 oak-md1-s2 kernel: #012Call Trace: Feb 21 11:39:20 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 11:39:20 oak-md1-s2 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Feb 21 11:39:20 oak-md1-s2 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Feb 21 11:39:20 oak-md1-s2 kernel: [] down_write+0x2d/0x3d Feb 21 11:39:20 oak-md1-s2 kernel: [] lod_qos_prep_create+0xaa4/0x17f0 [lod] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs] Feb 21 11:39:20 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 11:39:20 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 11:39:20 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 11:39:20 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 11:39:20 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 11:39:20 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 11:39:20 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:39:20 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 11:39:20 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 11:39:20 oak-md1-s2 kernel: Feb 21 11:39:20 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241960.103172 Feb 21 11:39:45 oak-md1-s2 kernel: LNet: Service thread pid 103187 was inactive for 460.57s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:39:45 oak-md1-s2 kernel: LNet: Skipped 4 previous similar messages Feb 21 11:39:45 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241985.103187 Feb 21 11:39:47 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241987.220680 Feb 21 11:39:49 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241989.220656 Feb 21 11:39:50 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519241990.103185 Feb 21 11:40:14 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff880cd55da100 x1591134459362640/t0(0) o101->152ae78b-0b64-865e-0fa5-0e12e2572d24@10.8.0.64@o2ib6:64/0 lens 880/3512 e 10 to 0 dl 1519242019 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:40:19 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff880dc02ce000 x1592341522909088/t0(0) o101->6591cb28-c82a-cd3d-7630-0ad8978b04bd@10.8.2.24@o2ib6:69/0 lens 896/3512 e 10 to 0 dl 1519242024 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:40:20 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff8804e3f71500 x1591082849842704/t0(0) o101->9543ffeb-99ef-8066-2f73-fc28979ae6db@10.9.101.17@o2ib4:70/0 lens 896/3512 e 10 to 0 dl 1519242025 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:40:20 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Feb 21 11:40:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 152ae78b-0b64-865e-0fa5-0e12e2572d24 (at 10.8.0.64@o2ib6) reconnecting Feb 21 11:40:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 152ae78b-0b64-865e-0fa5-0e12e2572d24 (at 10.8.0.64@o2ib6) Feb 21 11:40:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 11:40:22 oak-md1-s2 kernel: Lustre: 221412:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff880e99aa5d00 x1591135371159744/t0(0) o101->a30044b2-cda3-e500-9b4b-8e63aa6fcbb5@10.9.101.4@o2ib4:72/0 lens 896/3512 e 10 to 0 dl 1519242027 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:40:22 oak-md1-s2 kernel: Lustre: 221412:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Feb 21 11:40:24 oak-md1-s2 kernel: LNet: Service thread pid 103156 completed after 599.93s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:40:24 oak-md1-s2 kernel: LNet: Skipped 1 previous similar message Feb 21 11:40:36 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff8802807e6300 x1592238638306912/t0(0) o101->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:86/0 lens 944/3512 e 10 to 0 dl 1519242041 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:40:36 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Feb 21 11:41:17 oak-md1-s2 kernel: LNet: Service thread pid 220683 was inactive for 511.61s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:41:17 oak-md1-s2 kernel: LNet: Skipped 5 previous similar messages Feb 21 11:41:17 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519242077.220683 Feb 21 11:41:56 oak-md1-s2 kernel: LustreError: 220652:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241816, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880579201e00/0xec47b06a6b8ec0ae lrc: 3/1,0 mode: --/PR res: [0x200002f75:0xead5:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 220652 timeout: 0 lvb_type: 0 Feb 21 11:41:59 oak-md1-s2 kernel: Lustre: 220734:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff881b1fe2bc00 x1593031489007920/t0(0) o101->131d7936-e256-a1ec-d17e-352915fa15e4@10.9.114.6@o2ib4:169/0 lens 904/3512 e 3 to 0 dl 1519242124 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:41:59 oak-md1-s2 kernel: Lustre: 220734:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Feb 21 11:42:04 oak-md1-s2 kernel: Lustre: 102846:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:100s); client may timeout. req@ffff881e0b280f00 x1592238638302320/t150503064375(0) o101->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:69/0 lens 880/648 e 14 to 0 dl 1519242024 ref 1 fl Complete:/0/0 rc 0/0 Feb 21 11:42:04 oak-md1-s2 kernel: LNet: Service thread pid 103147 completed after 599.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:42:04 oak-md1-s2 kernel: Lustre: 102846:0:(service.c:2112:ptlrpc_server_handle_request()) Skipped 5 previous similar messages Feb 21 11:42:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ee1d0378-e033-452a-77a7-2c16b8413b0f (at 10.9.101.34@o2ib4) reconnecting Feb 21 11:42:04 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Feb 21 11:42:10 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519242130.199941 Feb 21 11:42:16 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519242136.103139 Feb 21 11:42:23 oak-md1-s2 kernel: Lustre: 220734:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff881d86227800 x1591143951852512/t0(0) o101->96cd5cb9-37a6-6cad-3e22-73bc9512afdf@10.9.104.56@o2ib4:193/0 lens 896/3512 e 3 to 0 dl 1519242148 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:42:23 oak-md1-s2 kernel: Lustre: 220734:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages Feb 21 11:43:27 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012 req@ffff880b58398f00 x1592938264491520/t0(0) o101->c1f5120b-2334-a0f7-38fe-67e4cf3ffbc2@10.8.1.29@o2ib6:257/0 lens 896/3512 e 2 to 0 dl 1519242212 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:43:27 oak-md1-s2 kernel: Lustre: 220677:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Feb 21 11:43:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client aa0141b0-d017-626c-c09d-b75d6daae2e4 (at 10.8.1.28@o2ib6) reconnecting Feb 21 11:43:33 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:43:42 oak-md1-s2 kernel: LNet: Service thread pid 220675 was inactive for 610.66s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 21 11:43:42 oak-md1-s2 kernel: LNet: Skipped 2 previous similar messages Feb 21 11:43:42 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519242222.220675 Feb 21 11:43:44 oak-md1-s2 kernel: Lustre: 103148:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:90s); client may timeout. req@ffff881859fb1200 x1593031492858784/t150503073610(0) o101->703ddb01-b17f-5bba-8d37-cf828621cf91@10.9.114.5@o2ib4:179/0 lens 896/648 e 3 to 0 dl 1519242134 ref 1 fl Complete:/0/0 rc 0/0 Feb 21 11:43:44 oak-md1-s2 kernel: LNet: Service thread pid 103172 completed after 676.39s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 11:43:44 oak-md1-s2 kernel: LNet: Skipped 12 previous similar messages Feb 21 11:43:44 oak-md1-s2 kernel: Lustre: 103148:0:(service.c:2112:ptlrpc_server_handle_request()) Skipped 10 previous similar messages Feb 21 11:44:53 oak-md1-s2 kernel: LustreError: 103140:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519241993, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8819e4a37800/0xec47b06a6b981aae lrc: 3/1,0 mode: --/PR res: [0x20000db45:0x16bae:0x0].0x0 bits 0x13 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103140 timeout: 0 lvb_type: 0 Feb 21 11:44:53 oak-md1-s2 kernel: LustreError: 103140:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message Feb 21 11:47:04 oak-md1-s2 kernel: LustreError: 102846:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519242124, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880e83ca4200/0xec47b06a6b9ee1f8 lrc: 3/1,0 mode: --/PR res: [0x20000db01:0xda33:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 102846 timeout: 0 lvb_type: 0 Feb 21 11:47:04 oak-md1-s2 kernel: LustreError: 102846:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message Feb 21 11:48:44 oak-md1-s2 kernel: LustreError: 103194:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519242224, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880dcceffa00/0xec47b06a6ba31bc5 lrc: 3/1,0 mode: --/PR res: [0x20000db2b:0xb4f2:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103194 timeout: 0 lvb_type: 0 Feb 21 11:50:20 oak-md1-s2 kernel: Lustre: 199942:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-63), not sending early reply#012 req@ffff880477ce0000 x1592341229697456/t0(0) o101->d5dc4bc6-1544-c3cc-b202-164df739673f@10.8.2.17@o2ib6:669/0 lens 888/3512 e 0 to 0 dl 1519242624 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:50:20 oak-md1-s2 kernel: Lustre: 199942:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Feb 21 11:50:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d5dc4bc6-1544-c3cc-b202-164df739673f (at 10.8.2.17@o2ib6) reconnecting Feb 21 11:50:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Feb 21 11:50:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5dc4bc6-1544-c3cc-b202-164df739673f (at 10.8.2.17@o2ib6) Feb 21 11:50:25 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Feb 21 11:53:07 oak-md1-s2 kernel: LustreError: 103144:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519242487, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880fc5939600/0xec47b06a6baff1d5 lrc: 3/1,0 mode: --/PR res: [0x20000aeaf:0x1703:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103144 timeout: 0 lvb_type: 0 Feb 21 11:53:07 oak-md1-s2 kernel: LustreError: 103144:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Feb 21 11:56:34 oak-md1-s2 kernel: Lustre: 103139:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012 req@ffff8804ada26900 x1591082849944640/t0(0) o101->9543ffeb-99ef-8066-2f73-fc28979ae6db@10.9.101.17@o2ib4:289/0 lens 888/3512 e 0 to 0 dl 1519242999 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 11:56:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9543ffeb-99ef-8066-2f73-fc28979ae6db (at 10.9.101.17@o2ib4) reconnecting Feb 21 11:56:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 11:57:04 oak-md1-s2 kernel: Lustre: 220672:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (667:400s); client may timeout. req@ffff880477ce0000 x1592341229697456/t150503191547(0) o101->d5dc4bc6-1544-c3cc-b202-164df739673f@10.8.2.17@o2ib6:669/0 lens 888/600 e 0 to 0 dl 1519242624 ref 1 fl Complete:/0/0 rc 0/0 Feb 21 12:00:12 oak-md1-s2 kernel: LustreError: 222596:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519242912, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8801c774b800/0xec47b06a6bcec82a lrc: 3/1,0 mode: --/PR res: [0x20000db45:0x16bae:0x0].0x0 bits 0x13 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 222596 timeout: 0 lvb_type: 0 Feb 21 12:00:12 oak-md1-s2 kernel: LustreError: 222596:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 6 previous similar messages Feb 21 12:02:04 oak-md1-s2 kernel: Lustre: 221821:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:325s); client may timeout. req@ffff8804ada26900 x1591082849944640/t150503247308(0) o101->9543ffeb-99ef-8066-2f73-fc28979ae6db@10.9.101.17@o2ib4:289/0 lens 888/648 e 0 to 0 dl 1519242999 ref 1 fl Complete:/0/0 rc 0/0 Feb 21 12:02:37 oak-md1-s2 kernel: Lustre: 103147:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-151), not sending early reply#012 req@ffff881dc5170600 x1591135466145888/t0(0) o101->613b765b-5cd9-fccb-0749-c6f75ef8aa6a@10.9.0.61@o2ib4:651/0 lens 704/3384 e 0 to 0 dl 1519243361 ref 2 fl Interpret:/0/0 rc 0/0 Feb 21 12:02:37 oak-md1-s2 kernel: Lustre: 103147:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Feb 21 12:04:13 oak-md1-s2 kernel: LNet: Service thread pid 220734 was inactive for 1203.93s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 21 12:04:13 oak-md1-s2 kernel: Pid: 220734, comm: mdt01_019 Feb 21 12:04:13 oak-md1-s2 kernel: #012Call Trace: Feb 21 12:04:13 oak-md1-s2 kernel: [] schedule+0x29/0x70 Feb 21 12:04:13 oak-md1-s2 kernel: [] rwsem_down_read_failed+0x10d/0x1a0 Feb 21 12:04:13 oak-md1-s2 kernel: [] call_rwsem_down_read_failed+0x18/0x30 Feb 21 12:04:13 oak-md1-s2 kernel: [] down_read+0x20/0x40 Feb 21 12:04:13 oak-md1-s2 kernel: [] lod_alloc_rr.constprop.18+0x22c/0x1000 [lod] Feb 21 12:04:13 oak-md1-s2 kernel: [] lod_qos_prep_create+0x12b9/0x17f0 [lod] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? qsd_op_begin+0xb0/0x4d0 [lquota] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? osd_otable_it_next+0x3a1/0x800 [osd_ldiskfs] Feb 21 12:04:13 oak-md1-s2 kernel: [] lod_prepare_create+0x298/0x3f0 [lod] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? osd_idc_find_and_init+0x7e/0x100 [osd_ldiskfs] Feb 21 12:04:13 oak-md1-s2 kernel: [] lod_declare_striped_create+0x1ee/0x970 [lod] Feb 21 12:04:13 oak-md1-s2 kernel: [] lod_declare_create+0x1e4/0x540 [lod] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdd_declare_create+0x53/0xe20 [mdd] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdd_create+0x7d9/0x1320 [mdd] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdt_reint_open+0x218c/0x31a0 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Feb 21 12:04:13 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Feb 21 12:04:13 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Feb 21 12:04:13 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 21 12:04:13 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Feb 21 12:04:13 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 12:04:13 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Feb 21 12:04:13 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Feb 21 12:04:13 oak-md1-s2 kernel: Feb 21 12:04:13 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1519243453.220734 Feb 21 12:04:28 oak-md1-s2 kernel: Lustre: Failing over oak-MDT0000 Feb 21 12:04:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.8.29.6@o2ib6 (stopping) Feb 21 12:04:28 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Feb 21 12:04:28 oak-md1-s2 kernel: LustreError: 102706:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881ee963a100 x1592931641746176/t0(0) o13->oak-OST0004-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:04:28 oak-md1-s2 kernel: LustreError: 102707:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881e168f0300 x1592931641746208/t0(0) o13->oak-OST000f-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:04:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.210.44.100@o2ib3 (stopping) Feb 21 12:04:28 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 102705:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881d9fc3d100 x1592931641747136/t0(0) o13->oak-OST004b-osc-MDT0000@10.0.2.105@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 102705:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages Feb 21 12:04:29 oak-md1-s2 kernel: Lustre: 103192:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:108s); client may timeout. req@ffff881dc5170600 x1591135466145888/t0(0) o101->613b765b-5cd9-fccb-0749-c6f75ef8aa6a@10.9.0.61@o2ib4:651/0 lens 704/536 e 0 to 0 dl 1519243361 ref 1 fl Complete:/0/0 rc -19/-19 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1100:ldlm_resource_complain()) mdt-oak-MDT0000_UUID: namespace resource [0x20000c380:0x3780:0x0].0x0 (ffff88102102f200) refcount nonzero (2) after lock cleanup; forcing cleanup. Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3780:0x0].0x0 (ffff88102102f200) refcount = 8 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1688:ldlm_resource_dump()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff880ff871b800/0xec47b06a6bcf9297 lrc: 2/0,1 mode: CW/CW res: [0x20000c380:0x3780:0x0].0x0 bits 0x2 rrc: 9 type: IBT flags: 0x40316400000000 nid: local remote: 0x0 expref: -99 pid: 220684 timeout: 0 lvb_type: 0 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2f:0x1afc5:0x0].0xa544118d (ffff8802c2685c80) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db28:0xb259:0x0].0x0 (ffff880efee99980) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3781:0x0].0x0 (ffff8818fbe5c300) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db35:0xc431:0x0].0xd83e99c (ffff8808cc3f8840) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3782:0x0].0x0 (ffff881efc604e40) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3783:0x0].0x0 (ffff88102102ec00) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2915:0x0].0x0 (ffff88088dbd83c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db30:0xb87a:0x0].0x5815ee5c (ffff88057b72f800) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2b:0xb4fb:0x0].0x2edbe7e0 (ffff8815ca2d8c00) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa28:0x5362:0x0].0x0 (ffff880d06ac20c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c370:0xb9a:0x0].0x0 (ffff880cfa7ab740) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2562:0x0].0x66643a (ffff880ad54d1740) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db36:0xc793:0x0].0x6f9245c6 (ffff8803926e69c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db32:0xde5d:0x0].0x0 (ffff880d3a68e0c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db01:0xda33:0x0].0x9ae88adb (ffff8815a0fa18c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200009b75:0x19b9c:0x0].0x0 (ffff881da5698d80) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200009c22:0xbf0:0x0].0x0 (ffff8805ad69d500) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3781:0x0].0xf3a61d91 (ffff881e8fc96180) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200002ea1:0x18572:0x0].0x0 (ffff8815ce376840) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db33:0xb1e0:0x0].0x0 (ffff88100c1a8e40) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c387:0x1420f:0x0].0x0 (ffff880747df7ec0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c370:0xb9a:0x0].0x152c6a10 (ffff880cfa7ab440) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200009b75:0x19b9c:0x0].0x8988fd8b (ffff881cbabe49c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c474:0xe50:0x0].0x2edbe7e0 (ffff8808c9711200) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa20:0x3085:0x0].0x0 (ffff880fdbd1b740) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000609e:0x1588:0x0].0x0 (ffff880cb519a9c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200008ee0:0xbc6d:0x0].0x7f914c59 (ffff88002108c000) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db45:0x16bae:0x0].0x36beed33 (ffff881b286c00c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2b:0xb4fb:0x0].0x0 (ffff8815ca2d8a80) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c473:0x436:0x0].0x0 (ffff880957108780) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db30:0xb87a:0x0].0x0 (ffff880c2a76b2c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000aeaf:0x1703:0x0].0xd28253f9 (ffff8815c8677200) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db29:0xcd32:0x0].0x2b9d35d4 (ffff8805f6706c00) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db33:0xb1e0:0x0].0xd83e99c (ffff880a40bfb2c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3780:0x0].0x14c422f (ffff88085f78dd40) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c474:0xe3d:0x0].0x0 (ffff88088fa4e0c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000b0ea:0x6d0:0x0].0xe3ffe3f8 (ffff880b6eda4600) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 102824:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.210.44.250@o2ib3 arrived at 1519243469 with bad export cookie 17025770882289818966 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000aeaf:0x1703:0x0].0x0 (ffff881e8fd6a3c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db01:0xda33:0x0].0x0 (ffff8815a0fa1e00) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2d:0x15e88:0x0].0x0 (ffff880bb5a81800) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3783:0x0].0x72cde134 (ffff8815f4642540) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000b0ea:0x6d0:0x0].0x0 (ffff880b6eda5140) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200002ea1:0x18572:0x0].0xc2de43aa (ffff8815ce376780) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000609e:0x1588:0x0].0xb8d35299 (ffff880ff773c3c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c474:0xe50:0x0].0x0 (ffff8808c9711440) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000e2b0:0x14b14:0x0].0x0 (ffff880602628180) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2562:0x0].0x5531d074 (ffff881db892d200) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa21:0x9c7:0x0].0x2edbe7e0 (ffff880cb519b140) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2c:0xb81d:0x0].0x0 (ffff880ac76c3b00) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db36:0xc793:0x0].0x0 (ffff88042d184000) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2e:0x999c:0x0].0x0 (ffff880f18ff7c80) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa28:0x6b43:0x0].0x676f6c5f (ffff8815c8baf140) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x377f:0x0].0xd538b54d (ffff880ed9222e40) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200008ee0:0xbc6d:0x0].0x0 (ffff880c11fd4b40) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x268c:0x0].0x6c713d4 (ffff880cb3b24cc0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2562:0x0].0x0 (ffff880aada0d740) refcount = 9 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db32:0xde5d:0x0].0x6f9245c6 (ffff880d3a68e9c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x3782:0x0].0xcb7cd953 (ffff8805ceaf4c00) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db31:0xe1a4:0x0].0x0 (ffff880e4b257d40) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db28:0xb259:0x0].0x1fd5ab57 (ffff880b80fba3c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000b0f6:0x3e3:0x0].0x0 (ffff880d70fb2180) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2915:0x0].0x6c713d4 (ffff88088dbd9ec0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000b0f6:0x3e3:0x0].0x277ab402 (ffff881cbabe4000) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db29:0xcd32:0x0].0x0 (ffff880d79ca0180) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db31:0xe1a4:0x0].0x5f16f586 (ffff880e4b257140) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c39d:0x444:0x0].0xee163afa (ffff8815ce3775c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2d:0x15e88:0x0].0xd419f726 (ffff8803fa387a40) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x200009c22:0xbf0:0x0].0x892517cb (ffff8805ad69c180) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2c:0xb81d:0x0].0x93adf796 (ffff880ac76c3440) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c39d:0x444:0x0].0x0 (ffff8815ce3769c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c473:0x436:0x0].0x43e5eb3a (ffff880a9b3ee6c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2a:0xd1bc:0x0].0x0 (ffff880100000000) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db35:0xc431:0x0].0x0 (ffff8808cc3f9380) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000e2b0:0x14b14:0x0].0x89f9fa23 (ffff8815a1314c00) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2a:0xd1bc:0x0].0x26a22f01 (ffff880100001440) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c379:0x66f:0x0].0x0 (ffff880ae8d06e40) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa21:0x9c7:0x0].0x0 (ffff880cb519b440) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c387:0x1420f:0x0].0xdbc26f53 (ffff881d839f7c80) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2562:0x0].0x5624d116 (ffff880be9a623c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa28:0x6b43:0x0].0x0 (ffff880acced4cc0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa20:0x3085:0x0].0x6f9245c6 (ffff880893417c80) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c474:0xe3d:0x0].0x43e5eb3a (ffff880860f8b2c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x2562:0x0].0x14e3a819 (ffff880be9a638c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000183f:0xf2d:0x0].0x0 (ffff881ca8217c80) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2e:0x999c:0x0].0x4dd5aba2 (ffff880f18ff7440) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000183f:0xf2d:0x0].0x35766f86 (ffff88064d2d2000) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db45:0x16bae:0x0].0x0 (ffff880b1b4483c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000db2f:0x1afc5:0x0].0x0 (ffff8802c26855c0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c379:0x66f:0x0].0x152c6a10 (ffff88064d2d35c0) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000fa28:0x5362:0x0].0xdc2853f0 (ffff880d06ac2600) refcount = 2 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000f271:0x268c:0x0].0x0 (ffff880cb3b25ec0) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x20000c380:0x377f:0x0].0x0 (ffff88102102ef00) refcount = 3 Feb 21 12:04:29 oak-md1-s2 kernel: LustreError: 223593:0:(ldlm_resource.c:1685:ldlm_resource_dump()) Granted locks (in reverse order): Feb 21 12:04:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.210.46.86@o2ib3 (stopping) Feb 21 12:04:29 oak-md1-s2 kernel: Lustre: Skipped 46 previous similar messages Feb 21 12:04:30 oak-md1-s2 kernel: LustreError: 102711:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880b0d5e3000 x1592931641747248/t0(0) o13->oak-OST000b-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:04:30 oak-md1-s2 kernel: LustreError: 102711:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages Feb 21 12:04:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.9.105.65@o2ib4 (stopping) Feb 21 12:04:31 oak-md1-s2 kernel: Lustre: Skipped 109 previous similar messages Feb 21 12:04:32 oak-md1-s2 kernel: LustreError: 102705:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881d9fc39e00 x1592931641747840/t0(0) o13->oak-OST000d-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:04:32 oak-md1-s2 kernel: LustreError: 102705:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 31 previous similar messages Feb 21 12:04:35 oak-md1-s2 kernel: LustreError: 112593:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.8.15.6@o2ib6 arrived at 1519243475 with bad export cookie 17025770887122163203 Feb 21 12:04:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.210.44.45@o2ib3 (stopping) Feb 21 12:04:36 oak-md1-s2 kernel: Lustre: Skipped 172 previous similar messages Feb 21 12:04:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.210.46.109@o2ib3 (stopping) Feb 21 12:04:44 oak-md1-s2 kernel: Lustre: Skipped 526 previous similar messages Feb 21 12:04:54 oak-md1-s2 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-oak-MDT0000_UUID namespace with 60 resources in use, (rc=-110) Feb 21 12:05:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.12.0.22@o2ib (stopping) Feb 21 12:05:01 oak-md1-s2 kernel: Lustre: Skipped 406 previous similar messages Feb 21 12:05:19 oak-md1-s2 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-oak-MDT0000_UUID namespace with 60 resources in use, (rc=-110) Feb 21 12:05:24 oak-md1-s2 kernel: LustreError: 103142:0:(lod_qos.c:208:lod_statfs_and_check()) oak-MDT0000-mdtlov: statfs: rc = -108 Feb 21 12:05:24 oak-md1-s2 kernel: Lustre: 220734:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:520s); client may timeout. req@ffff880eaf581e00 x1593032013427072/t0(0) o101->988dff17-04a2-fda9-e7f9-ee1aff163822@10.9.101.32@o2ib4:294/0 lens 904/544 e 0 to 0 dl 1519243004 ref 1 fl Complete:/0/0 rc -19/-19 Feb 21 12:05:24 oak-md1-s2 kernel: LNet: Service thread pid 220734 completed after 1274.78s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 21 12:05:24 oak-md1-s2 kernel: LNet: Skipped 11 previous similar messages Feb 21 12:05:26 oak-md1-s2 kernel: LustreError: 223691:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880afe5f2a00 x1592931641748224/t0(0) o101->oak-MDT0000-lwp-MDT0000@0@lo:23/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 Feb 21 12:05:26 oak-md1-s2 kernel: LustreError: 223691:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 19 previous similar messages Feb 21 12:05:26 oak-md1-s2 kernel: LustreError: 223691:0:(qsd_reint.c:56:qsd_reint_completion()) oak-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x1010000:0x0], rc:-5 Feb 21 12:05:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.9.104.28@o2ib4 (stopping) Feb 21 12:05:34 oak-md1-s2 kernel: Lustre: Skipped 148 previous similar messages Feb 21 12:06:24 oak-md1-s2 kernel: Lustre: server umount oak-MDT0000 complete Feb 21 12:06:52 oak-md1-s2 kernel: LNetError: 126679:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.204@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Feb 21 12:06:53 oak-md1-s2 kernel: LNetError: 126679:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.205@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Feb 21 12:06:53 oak-md1-s2 kernel: LNetError: 126679:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 5 previous similar messages Feb 21 12:06:54 oak-md1-s2 kernel: LNet: Removed LNI 10.0.2.52@o2ib5 Feb 21 12:07:03 oak-md1-s2 kernel: LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 Feb 21 12:07:03 oak-md1-s2 kernel: alg: No test for adler32 (adler32-zlib) Feb 21 12:07:03 oak-md1-s2 kernel: alg: No test for crc32 (crc32-table) Feb 21 12:07:03 oak-md1-s2 kernel: Lustre: Lustre: Build Version: 2.10.3_srcc1 Feb 21 12:07:04 oak-md1-s2 kernel: LNet: Using FMR for registration Feb 21 12:07:04 oak-md1-s2 kernel: LNet: Added LNI 10.0.2.52@o2ib5 [8/256/0/180] Feb 21 12:07:05 oak-md1-s2 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Feb 21 12:07:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Feb 21 12:07:07 oak-md1-s2 kernel: Lustre: oak-MDD0000: changelog on Feb 21 12:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-229]@o2ib5 10.0.2.[232-235]@o2ib5 10.0.2.[240-241]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Feb 21 12:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Feb 21 12:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Will be in recovery for at least 2:30, or until 1263 clients reconnect Feb 21 12:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c7d227ff-5bb7-6d8f-98d5-2f28c9054028 (at 10.210.47.63@o2ib3) Feb 21 12:07:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7501b480-0e38-9a56-f320-ecb2b7074c53 (at 10.9.101.41@o2ib4) Feb 21 12:07:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Feb 21 12:07:11 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4730365 seconds Feb 21 12:07:11 oak-md1-s2 kernel: Lustre: 224039:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1519243627/real 1519243631] req@ffff880963e22700 x1593042401231584/t0(0) o8->oak-OST000e-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1519243632 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 21 12:07:12 oak-md1-s2 kernel: Lustre: 224039:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1519243627/real 0] req@ffff880963e24200 x1593042401233248/t0(0) o8->oak-OST0053-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1519243632 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Feb 21 12:07:12 oak-md1-s2 kernel: Lustre: 224039:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 21 12:07:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72b29f82-4d37-17f9-24b2-9d652861e43f (at 10.12.4.83@o2ib) Feb 21 12:07:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Feb 21 12:07:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c532732-11b4-1dfc-e91d-d8f95cc5200f (at 10.9.102.50@o2ib4) Feb 21 12:07:14 oak-md1-s2 kernel: Lustre: Skipped 204 previous similar messages Feb 21 12:07:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 130a9a17-9761-777e-d635-6c04f1e0226b (at 10.210.46.177@o2ib3) Feb 21 12:07:18 oak-md1-s2 kernel: Lustre: Skipped 354 previous similar messages Feb 21 12:07:23 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4730377 seconds Feb 21 12:07:23 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 12 previous similar messages Feb 21 12:07:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 088b5d39-9a16-2e43-742a-38e3bebb7088 (at 10.9.102.59@o2ib4) Feb 21 12:07:26 oak-md1-s2 kernel: Lustre: Skipped 480 previous similar messages Feb 21 12:07:36 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4730390 seconds Feb 21 12:07:36 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 12 previous similar messages Feb 21 12:07:49 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4730403 seconds Feb 21 12:07:49 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 8 previous similar messages Feb 21 12:07:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Feb 21 12:07:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Feb 21 12:07:50 oak-md1-s2 kernel: Lustre: Skipped 252 previous similar messages Feb 21 12:07:50 oak-md1-s2 kernel: Lustre: Skipped 294 previous similar messages Feb 21 12:08:01 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4 seconds Feb 21 12:08:01 oak-md1-s2 kernel: LNet: 224026:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 21 12:19:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:53. If you do not want to wait more, please abort the recovery by force. Feb 21 12:19:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72b29f82-4d37-17f9-24b2-9d652861e43f (at 10.12.4.83@o2ib) Feb 21 12:19:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Feb 21 12:19:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:53. If you do not want to wait more, please abort the recovery by force. Feb 21 12:19:53 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:52. If you do not want to wait more, please abort the recovery by force. Feb 21 12:19:54 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Feb 21 12:19:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:50. If you do not want to wait more, please abort the recovery by force. Feb 21 12:19:56 oak-md1-s2 kernel: Lustre: Skipped 25 previous similar messages Feb 21 12:20:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:41. If you do not want to wait more, please abort the recovery by force. Feb 21 12:20:04 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Feb 21 12:20:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b1a87fee-dc49-f6fb-bad8-540239838301 (at 10.12.4.27@o2ib) Feb 21 12:20:04 oak-md1-s2 kernel: Lustre: Skipped 62 previous similar messages Feb 21 12:20:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery already passed deadline 1:32. If you do not want to wait more, please abort the recovery by force. Feb 21 12:20:14 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Feb 21 12:21:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: recovery is timed out, evict stale exports Feb 21 12:21:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: disconnecting 1 stale clients Feb 21 12:21:46 oak-md1-s2 kernel: Lustre: 224323:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Feb 21 12:21:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery over after 14:38, of 1263 clients 1262 recovered and 1 was evicted. Feb 21 12:44:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17dab95c-3765-c601-9b27-f4222ca95dab (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e323000, cur 1519245868 expire 1519245718 last 1519245641 Feb 21 13:18:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881038effc00, cur 1519247904 expire 1519247754 last 1519247677 Feb 21 13:27:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) Feb 21 13:27:37 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages