Oct 14 16:24:05 f1-mgs1 kernel: [2605266.836639] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:24:39 f1-mgs1 kernel: [2605300.175778] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:30:12 f1-oss1d8 kernel: [2604418.085966] LustreError: 10030:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.041236] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.065422] Lustre: Skipped 3 previous similar messages Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.076384] LustreError: 99512:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.125308] LustreError: 99512:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 16:33:11 f1-oss1d8 kernel: [2604597.299596] LustreError: 38769:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 16:33:11 f1-oss1d8 kernel: [2604597.348347] LustreError: 38769:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 16:39:06 f1-mds1 kernel: [2606181.972594] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:39:08 f1-mds1 kernel: [2606184.473277] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:39:42 f1-mds1 kernel: [2606217.952414] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:40:07 f1-mds-spare01 kernel: [2606203.309901] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:40:40 f1-mds-spare01 kernel: [2606236.533759] ACPI: No handler for Region [POWR] (ffff884053eb24b0) [IPMI] Oct 14 16:43:41 f1-oss1d8 kernel: [2605227.264660] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 16:43:41 f1-oss1d8 kernel: [2605227.264669] LustreError: 99504:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e9:0x0]: -115 Oct 14 16:43:41 f1-oss1d8 kernel: [2605227.264673] LustreError: 99504:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 16:43:41 f1-oss1d8 kernel: [2605227.365222] Lustre: Skipped 3 previous similar messages Oct 14 16:43:45 f1-oss1d8 kernel: [2605231.557780] LustreError: 42431:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 16:43:45 f1-oss1d8 kernel: [2605231.607418] LustreError: 42431:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 16:46:14 f1-mds1 kernel: [2606610.960972] LustreError: 18064:0:(tgt_lastrcvd.c:813:tgt_last_rcvd_update()) f1-MDT0000: trying to overwrite bigger transno:on-disk: 353968921468, new: 353968921467 replay: 0. see LU-617. Oct 14 16:47:54 f1-oss1d8 kernel: [2605480.651396] LustreError: 44253:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 16:50:10 f1-oss1a1 kernel: [2605617.201675] LustreError: 87273:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 16:52:20 f1-oss1c2 kernel: [2592958.071456] Lustre: 22518:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444840973/real 1444840973] req@ffff880d2e595400 x1512308858665024/t0(0) o106->f1-OST0061@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444841540 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 16:52:20 f1-oss1c8 kernel: [2592968.995539] Lustre: 2635:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444840973/real 1444840973] req@ffff880d01dd0c00 x1512308951545672/t0(0) o106->f1-OST00b7@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444841540 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 16:52:20 f1-oss1b8 kernel: [2605751.539213] Lustre: 14273:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444840973/real 1444840973] req@ffff880dae883400 x1512295974540968/t0(0) o106->f1-OST000f@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444841540 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 16:52:20 f1-oss1e3 kernel: [2592964.132974] Lustre: 22446:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444840973/real 1444840973] req@ffff880d1ac7c000 x1512308988321116/t0(0) o106->f1-OST00ea@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444841540 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 16:52:48 f1-oss1b8 kernel: [2605779.543569] Lustre: 98790:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Oct 14 16:52:48 f1-oss1b8 kernel: [2605779.543571] req@ffff881009dc7850 x1512380036519996/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/400 e 12 to 0 dl 1444841573 ref 2 fl Interpret:/0/0 rc 0/0 Oct 14 16:52:48 f1-oss1c2 kernel: [2592986.079768] Lustre: 16276:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Oct 14 16:52:48 f1-oss1c2 kernel: [2592986.079769] req@ffff881041d43050 x1512380036520004/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/400 e 12 to 0 dl 1444841573 ref 2 fl Interpret:/0/0 rc 0/0 Oct 14 16:52:48 f1-oss1e3 kernel: [2592992.140376] Lustre: 22495:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Oct 14 16:52:48 f1-oss1e3 kernel: [2592992.140378] req@ffff880ff5097850 x1512380036520000/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/400 e 12 to 0 dl 1444841573 ref 2 fl Interpret:/0/0 rc 0/0 Oct 14 16:52:48 f1-oss1c8 kernel: [2592997.005903] Lustre: 17104:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Oct 14 16:52:48 f1-oss1c8 kernel: [2592997.005904] req@ffff88102553f850 x1512380036519992/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/400 e 12 to 0 dl 1444841573 ref 2 fl Interpret:/0/0 rc 0/0 Oct 14 16:54:15 f1-oss1d8 kernel: [2605861.500088] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 16:54:15 f1-oss1d8 kernel: [2605861.524164] Lustre: Skipped 4 previous similar messages Oct 14 16:54:15 f1-oss1d8 kernel: [2605861.543668] LustreError: 90090:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 16:54:15 f1-oss1d8 kernel: [2605861.584065] LustreError: 90090:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 16:54:19 f1-oss1d8 kernel: [2605865.827819] LustreError: 47184:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 16:54:19 f1-oss1d8 kernel: [2605865.876932] LustreError: 47184:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 16:56:06 f1-oss1a6 kernel: [2605962.474839] LustreError: 12729:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 16:57:51 f1-oss1d3 kernel: [2606063.788503] LustreError: 14992:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 17:01:47 f1-oss1c2 kernel: [2593525.358066] Lustre: 22518:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444841540/real 1444841540] req@ffff880d2e595400 x1512308858665024/t0(0) o106->f1-OST0061@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444842107 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 17:01:47 f1-oss1c8 kernel: [2593536.291241] Lustre: 2635:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444841540/real 1444841540] req@ffff880d01dd0c00 x1512308951545672/t0(0) o106->f1-OST00b7@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444842107 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 17:01:47 f1-oss1b8 kernel: [2606318.832182] Lustre: 14273:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444841540/real 1444841540] req@ffff880dae883400 x1512295974540968/t0(0) o106->f1-OST000f@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444842107 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 17:01:47 f1-oss1e3 kernel: [2593531.430728] Lustre: 22446:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1444841540/real 1444841540] req@ffff880d1ac7c000 x1512308988321116/t0(0) o106->f1-OST00ea@1192@gni1:15/16 lens 296/280 e 0 to 1 dl 1444842107 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.302313] LNet: Service thread pid 22518 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.353779] LNet: Skipped 1 previous similar message Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.230733] LNet: Service thread pid 2635 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.282506] Pid: 2635, comm: ll_ost03_053 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.365213] LNet: Service thread pid 22446 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.417409] Pid: 22446, comm: ll_ost03_010 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.427322] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.427322] Call Trace: Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.372709] Pid: 22518, comm: ll_ost03_022 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.300579] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.300579] Call Trace: Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.383471] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.383471] Call Trace: Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.402569] [] schedule_timeout+0x192/0x2e0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.311369] [] schedule_timeout+0x192/0x2e0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.331580] [] ? process_timeout+0x0/0x10 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.446423] [] schedule_timeout+0x192/0x2e0 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.466446] [] ? process_timeout+0x0/0x10 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.422359] [] ? process_timeout+0x0/0x10 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.351426] [] ptlrpc_set_wait+0x2da/0x860 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.371712] [] ? default_wake_function+0x0/0x20 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.486514] [] ptlrpc_set_wait+0x2da/0x860 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.506932] [] ? default_wake_function+0x0/0x20 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.442236] [] ptlrpc_set_wait+0x2da/0x860 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.462775] [] ? default_wake_function+0x0/0x20 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.391967] [] ? ldlm_work_gl_ast_lock+0x0/0x290 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.421253] [] ldlm_run_ast_work+0x1bb/0x470 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.527230] [] ? ldlm_work_gl_ast_lock+0x0/0x290 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.482948] [] ? ldlm_work_gl_ast_lock+0x0/0x290 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.503833] [] ldlm_run_ast_work+0x1bb/0x470 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.556573] [] ldlm_run_ast_work+0x1bb/0x470 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.441283] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.576904] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.532501] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.552958] [] ofd_intent_policy+0x53e/0x7f0 [ofd] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.461817] [] ofd_intent_policy+0x53e/0x7f0 [ofd] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.597337] [] ofd_intent_policy+0x53e/0x7f0 [ofd] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.625965] [] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.482310] [] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.573518] [] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.602282] [] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.511082] [] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.646368] [] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.532277] [] ldlm_handle_enqueue+0x5e/0x70 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.667394] [] ldlm_handle_enqueue+0x5e/0x70 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.623405] [] ldlm_handle_enqueue+0x5e/0x70 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.560710] [] ? ldlm_server_completion_ast+0x0/0x770 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.582011] [] ? ost_blocking_ast+0x0/0x1020 [ost] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.695788] [] ? ldlm_server_completion_ast+0x0/0x770 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.717220] [] ? ost_blocking_ast+0x0/0x1020 [ost] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.643484] [] ? ldlm_server_completion_ast+0x0/0x770 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.673246] [] ? ost_blocking_ast+0x0/0x1020 [ost] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.601830] [] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.631262] [] ost_handle+0x1d1d/0x44f0 [ost] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.737140] [] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.766399] [] ost_handle+0x1d1d/0x44f0 [ost] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.693117] [] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.722471] [] ost_handle+0x1d1d/0x44f0 [ost] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.650946] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.672312] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.786393] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.742141] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.807640] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.836988] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.701701] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.763620] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.803429] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.721356] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.856939] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.877139] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.822316] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.742050] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.770788] [] ? __wake_up_common+0x59/0x90 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.906229] [] ? __wake_up_common+0x59/0x90 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.842950] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.863410] [] ? __wake_up_common+0x59/0x90 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.790793] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.926283] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.883462] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.811000] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.831418] [] kthread+0x9e/0xc0 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.946651] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.966984] [] kthread+0x9e/0xc0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.850808] [] child_rip+0xa/0x20 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.903679] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.932361] [] kthread+0x9e/0xc0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.862264] [] ? kthread+0x0/0xc0 Oct 14 17:02:53 f1-oss1e3 kernel: [2593597.986477] [] child_rip+0xa/0x20 Oct 14 17:02:53 f1-oss1e3 kernel: [2593598.005882] [] ? kthread+0x0/0xc0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.881570] [] ? child_rip+0x0/0x20 Oct 14 17:02:53 f1-oss1e3 kernel: [2593598.017153] [] ? child_rip+0x0/0x20 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.943694] [] child_rip+0xa/0x20 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.963268] [] ? kthread+0x0/0xc0 Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.901186] Oct 14 17:02:53 f1-oss1c8 kernel: [2593602.902654] LustreError: dumping log to /tmp/lustre-log.1444842173.2635 Oct 14 17:02:53 f1-oss1e3 kernel: [2593598.036687] Oct 14 17:02:53 f1-oss1e3 kernel: [2593598.045914] LustreError: dumping log to /tmp/lustre-log.1444842173.22446 Oct 14 17:02:53 f1-oss1c2 kernel: [2593591.982762] [] ? child_rip+0x0/0x20 Oct 14 17:02:53 f1-oss1c2 kernel: [2593592.002401] Oct 14 17:02:53 f1-oss1c2 kernel: [2593592.003733] LustreError: dumping log to /tmp/lustre-log.1444842173.22518 Oct 14 17:03:31 f1-mds1 kernel: [2607648.250289] Lustre: f1-MDT0000: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f750fd800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-mgs1 kernel: [2607634.156742] Lustre: MGS: haven't heard from client 5b18cf84-d1d1-2b95-4fa1-41e078bda144 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff884033119400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b2 kernel: [2606426.943076] Lustre: f1-OST00f9: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100ed13000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b2 kernel: [2606426.999379] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-mds1 kernel: [2607648.274580] LustreError: 17635:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) ### client (nid 1192@gni1) returned 0 from blocking AST ns: mdt-f1-MDT0000_UUID lock: ffff883e328b5340/0xe17d9cc3d683df1 lrc: 4/0,0 mode: PR/PR res: [0x200001c56:0x70c4:0x0].0 bits 0x13 rrc: 8 type: IBT flags: 0x60200000000020 nid: 1192@gni1 remote: 0xc9fdbc1ac7df8945 expref: 132 pid: 17593 timeout: 6900126313 lvb_type: 0 Oct 14 17:03:31 f1-oss1e3 kernel: [2593635.366744] Lustre: f1-OST009a: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ff8115c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a8 kernel: [2606421.198438] Lustre: f1-OST002f: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88104998a800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c6 kernel: [2593633.872897] Lustre: f1-OST00b5: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100145c400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c6 kernel: [2593633.936395] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1c5 kernel: [2593636.089867] Lustre: f1-OST008c: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881008886000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a5 kernel: [2606418.276444] Lustre: f1-OST0004: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88101c766400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a7 kernel: [2606417.726538] Lustre: f1-OST0006: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880fe795c800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a7 kernel: [2606417.792121] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b8 kernel: [2606422.773175] Lustre: f1-OST000f: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100cd1dc00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b8 kernel: [2606422.837977] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d7 kernel: [2606408.677222] Lustre: f1-OST001e: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ff7efa000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d1 kernel: [2606413.472650] Lustre: f1-OST0018: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881028315400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e2 kernel: [2593633.540146] Lustre: f1-OST0111: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881008d8dc00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d1 kernel: [2606413.532246] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b4 kernel: [2606396.692944] Lustre: f1-OST000b: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ffca27400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d8 kernel: [2606417.707657] Lustre: f1-OST001f: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881001fd3000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b4 kernel: [2606396.761734] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b6 kernel: [2606422.613538] Lustre: f1-OST00ad: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881042eb1c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d8 kernel: [2606417.774822] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b6 kernel: [2606422.677854] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e5 kernel: [2593632.680383] Lustre: f1-OST0114: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881007b84800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e5 kernel: [2593632.748996] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b7 kernel: [2606427.364540] Lustre: f1-OST000e: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ffdcf2c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.310958] Lustre: f1-OST0089: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881028952400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b7 kernel: [2606427.427973] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.377418] Lustre: Skipped 1 previous similar message Oct 14 17:03:31 f1-oss1d3 kernel: [2606403.936207] Lustre: f1-OST001a: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881030579c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e6 kernel: [2593636.088829] Lustre: f1-OST00ed: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810180eb800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e6 kernel: [2593636.151814] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d5 kernel: [2606405.179794] Lustre: f1-OST0094: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100af11000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.242570] Lustre: f1-OST0107: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100b87ec00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d5 kernel: [2606405.243507] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e4 kernel: [2593638.974222] Lustre: f1-OST004b: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881007696400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.305955] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e4 kernel: [2593639.038359] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1b1 kernel: [2596320.598395] Lustre: f1-OST0080: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88101b80a400, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c7 kernel: [2593624.903142] Lustre: f1-OST0016: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810022c1000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b1 kernel: [2596320.668343] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a1 kernel: [2606418.020483] Lustre: f1-OST00f0: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100bce6c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c7 kernel: [2593624.968210] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a3 kernel: [2606435.751582] Lustre: f1-OST0002: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ffbe77800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b5 kernel: [2606418.583819] Lustre: f1-OST000c: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88103fc2c800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a3 kernel: [2606435.817018] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d6 kernel: [2606430.446208] Lustre: f1-OST0045: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881009f83c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b5 kernel: [2606418.649384] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d6 kernel: [2606430.511796] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d2 kernel: [2606417.069842] Lustre: f1-OST00b9: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ffa8e1000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d2 kernel: [2606417.135140] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a8 kernel: [2606421.263106] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a2 kernel: [2606420.752674] Lustre: f1-OST0001: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810084d1c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a2 kernel: [2606420.819775] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1c1 kernel: [2593631.369496] Lustre: f1-OST0010: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881008b19800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c4 kernel: [2593636.737569] Lustre: f1-OST0013: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810089c6c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1c4 kernel: [2593636.805174] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e7 kernel: [2593639.304531] Lustre: f1-OST004e: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ff7d8c000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a4 kernel: [2606406.675904] Lustre: f1-OST007b: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881047f98000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e7 kernel: [2593639.372514] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a4 kernel: [2606406.746117] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e1 kernel: [2593619.268576] Lustre: f1-OST0020: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880fff3dcc00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e1 kernel: [2593619.338152] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1c3 kernel: [2593637.133311] Lustre: f1-OST0012: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881007977c00, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b3 kernel: [2606414.157088] Lustre: f1-OST000a: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88104553c000, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1b3 kernel: [2606414.227179] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1c3 kernel: [2593637.203317] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1a6 kernel: [2606406.755056] Lustre: f1-OST0005: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88100a160800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1a6 kernel: [2606406.825312] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1d4 kernel: [2606410.708088] Lustre: f1-OST0043: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810094e6800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1d4 kernel: [2606410.778497] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e8 kernel: [2593612.225309] Lustre: f1-OST004f: haven't heard from client 1f5ae33d-9529-3029-ceba-4d159b02b355 (at 1192@gni1) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ff6834800, cur 1444842211 expire 1444841311 last 1444840859 Oct 14 17:03:31 f1-oss1e8 kernel: [2593612.296112] Lustre: Skipped 6 previous similar messages Oct 14 17:03:31 f1-oss1e3 kernel: [2593635.431501] LustreError: 22446:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) ### client (nid 1192@gni1) returned 0 from glimpse AST ns: filter-f1-OST00ea_UUID lock: ffff880957a411c0/0xd7a28fcfd84956d2 lrc: 5/0,0 mode: PW/PW res: [0x519679b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40000000000000 nid: 1192@gni1 remote: 0xc9fdbc1ac7df8b67 expref: 4 pid: 22510 timeout: 0 lvb_type: 0 Oct 14 17:03:31 f1-oss1b8 kernel: [2606422.848172] LustreError: 14273:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) ### client (nid 1192@gni1) returned 0 from glimpse AST ns: filter-f1-OST000f_UUID lock: ffff8809c75f43c0/0x9a76d8a2c745cd85 lrc: 4/0,0 mode: PW/PW res: [0x51db9dc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40000000000000 nid: 1192@gni1 remote: 0xc9fdbc1ac7df8b60 expref: 6 pid: 98813 timeout: 0 lvb_type: 0 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.316427] LustreError: 2635:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) ### client (nid 1192@gni1) returned 0 from glimpse AST ns: filter-f1-OST00b7_UUID lock: ffff8809a25fddc0/0x680b3733cb2cf638 lrc: 2/0,0 mode: --/PW res: [0x50f14c6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x44801000000000 nid: 1192@gni1 remote: 0xc9fdbc1ac7df8b59 expref: 5 pid: 16647 timeout: 0 lvb_type: 0 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.446857] LustreError: 2635:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) Skipped 1 previous similar message Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.387913] LustreError: 22518:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) ### client (nid 1192@gni1) returned 0 from glimpse AST ns: filter-f1-OST0061_UUID lock: ffff8810169e3340/0x742147666934c051 lrc: 6/0,0 mode: PW/PW res: [0x5204d67:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x50801000000000 nid: 1192@gni1 remote: 0xc9fdbc1ac7df8b6e expref: 4 pid: 22533 timeout: 0 lvb_type: 0 Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.519378] LustreError: 22518:0:(ldlm_lockd.c:666:ldlm_handle_ast_error()) Skipped 1 previous similar message Oct 14 17:03:31 f1-oss1e3 kernel: [2593635.553479] Lustre: 22446:0:(service.c:2108:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:638s); client may timeout. req@ffff880ff5097850 x1512380036520000/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/368 e 12 to 0 dl 1444841573 ref 1 fl Complete:/0/0 rc 301/301 Oct 14 17:03:31 f1-oss1e3 kernel: [2593635.652227] LNet: Service thread pid 22446 completed after 1238.27s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Oct 14 17:03:31 f1-oss1b8 kernel: [2606422.979709] Lustre: 14273:0:(service.c:2108:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:638s); client may timeout. req@ffff881009dc7850 x1512380036519996/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/368 e 12 to 0 dl 1444841573 ref 1 fl Complete:/0/0 rc 301/301 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.475881] Lustre: 2635:0:(service.c:2108:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:638s); client may timeout. req@ffff88102553f850 x1512380036519992/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/368 e 12 to 0 dl 1444841573 ref 1 fl Complete:/0/0 rc 301/301 Oct 14 17:03:31 f1-oss1c8 kernel: [2593640.567777] LNet: Service thread pid 2635 completed after 1238.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.556899] Lustre: 22518:0:(service.c:2108:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:638s); client may timeout. req@ffff881041d43050 x1512380036520004/t0(0) o101->9610b23e-4027-8b7a-5a18-fde7c7c40eab@762@gni1:0/0 lens 328/368 e 12 to 0 dl 1444841573 ref 1 fl Complete:/0/0 rc 301/301 Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.648968] LNet: Service thread pid 22518 completed after 1238.33s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Oct 14 17:03:31 f1-oss1b8 kernel: [2606423.069945] LNet: Service thread pid 14273 completed after 1238.28s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Oct 14 17:03:31 f1-oss1c2 kernel: [2593629.707308] LNet: Skipped 11 previous similar messages Oct 14 17:04:49 f1-oss1d8 kernel: [2606495.754591] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 17:04:49 f1-oss1d8 kernel: [2606495.781321] LustreError: 99510:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 17:04:49 f1-oss1d8 kernel: [2606495.781324] LustreError: 99510:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 17:04:49 f1-oss1d8 kernel: [2606495.853956] Lustre: Skipped 1 previous similar message Oct 14 17:04:53 f1-oss1d8 kernel: [2606500.078861] LustreError: 50744:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:04:53 f1-oss1d8 kernel: [2606500.126411] LustreError: 50744:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 17:09:24 f1-oss1d1 kernel: [2606767.117784] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 55@gni2 ns: filter-f1-OST00b8_UUID lock: ffff880c2a2a0ac0/0xa0126f4fb03102e1 lrc: 3/0,0 mode: PW/PW res: [0x5091b83:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000000000020 nid: 55@gni2 remote: 0x8f8868f52e691da5 expref: 4 pid: 97586 timeout: 6900472931 lvb_type: 0 Oct 14 17:09:24 f1-oss1c8 kernel: [2593993.998193] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 52@gni2 ns: filter-f1-OST0017_UUID lock: ffff88038ca8fd00/0x680b3733cbb5beca lrc: 3/0,0 mode: PR/PR res: [0x4f53240:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000000000020 nid: 52@gni2 remote: 0x381a6499359c8d84 expref: 4 pid: 22327 timeout: 6887695822 lvb_type: 1 Oct 14 17:11:24 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 17:15:23 f1-oss1d8 kernel: [2607130.005031] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 17:15:23 f1-oss1d8 kernel: [2607130.010973] LustreError: 90090:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 17:15:23 f1-oss1d8 kernel: [2607130.010977] LustreError: 90090:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 17:15:23 f1-oss1d8 kernel: [2607130.103109] Lustre: Skipped 6 previous similar messages Oct 14 17:15:27 f1-oss1d8 kernel: [2607134.315526] LustreError: 54718:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:15:27 f1-oss1d8 kernel: [2607134.365676] LustreError: 54718:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 17:25:57 f1-oss1d8 kernel: [2607764.211311] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 17:25:57 f1-oss1d8 kernel: [2607764.211320] LustreError: 99519:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e3:0x0]: -115 Oct 14 17:25:57 f1-oss1d8 kernel: [2607764.211323] LustreError: 99519:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 17:25:57 f1-oss1d8 kernel: [2607764.312221] Lustre: Skipped 2 previous similar messages Oct 14 17:26:01 f1-oss1d8 kernel: [2607768.865940] LustreError: 58218:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:26:01 f1-oss1d8 kernel: [2607768.915311] LustreError: 58218:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 17:36:31 f1-oss1d8 kernel: [2608398.456434] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 17:36:31 f1-oss1d8 kernel: [2608398.456780] LustreError: 90014:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e3:0x0]: -115 Oct 14 17:36:31 f1-oss1d8 kernel: [2608398.456783] LustreError: 90014:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 17:36:31 f1-oss1d8 kernel: [2608398.562007] Lustre: Skipped 1 previous similar message Oct 14 17:36:35 f1-oss1d8 kernel: [2608402.744296] LustreError: 61795:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:36:35 f1-oss1d8 kernel: [2608402.794353] LustreError: 61795:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 17:47:05 f1-oss1d8 kernel: [2609032.716445] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 17:47:05 f1-oss1d8 kernel: [2609032.740915] Lustre: Skipped 1 previous similar message Oct 14 17:47:05 f1-oss1d8 kernel: [2609032.741356] LustreError: 59369:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e9:0x0]: -115 Oct 14 17:47:05 f1-oss1d8 kernel: [2609032.741359] LustreError: 59369:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 17:47:09 f1-oss1d8 kernel: [2609037.028298] LustreError: 65640:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:47:09 f1-oss1d8 kernel: [2609037.073705] LustreError: 65640:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 17:57:39 f1-oss1d8 kernel: [2609666.908474] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 17:57:39 f1-oss1d8 kernel: [2609666.931333] Lustre: Skipped 1 previous similar message Oct 14 17:57:39 f1-oss1d8 kernel: [2609666.936876] LustreError: 59387:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 17:57:39 f1-oss1d8 kernel: [2609666.936879] LustreError: 59387:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 17:57:43 f1-oss1d8 kernel: [2609671.214078] LustreError: 69437:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 17:57:43 f1-oss1d8 kernel: [2609671.262718] LustreError: 69437:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 18:06:48 f1-oss1a6 kernel: [2610206.074690] LustreError: 94346:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 18:08:13 f1-oss1d8 kernel: [2610301.148350] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 18:08:13 f1-oss1d8 kernel: [2610301.148360] LustreError: 121536:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 18:08:13 f1-oss1d8 kernel: [2610301.148363] LustreError: 121536:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 18:08:13 f1-oss1d8 kernel: [2610301.250040] Lustre: Skipped 5 previous similar messages Oct 14 18:08:17 f1-oss1d8 kernel: [2610305.460365] LustreError: 72982:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 18:08:17 f1-oss1d8 kernel: [2610305.512397] LustreError: 72982:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 18:11:23 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 18:15:00 f1-oss1a1 kernel: [2610709.448579] LustreError: 97529:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 18:15:05 f1-oss1d4 kernel: [2610706.503446] LustreError: 62515:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 18:18:47 f1-oss1d8 kernel: [2610935.375853] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 18:18:47 f1-oss1d8 kernel: [2610935.398957] Lustre: Skipped 1 previous similar message Oct 14 18:18:47 f1-oss1d8 kernel: [2610935.409891] LustreError: 1167:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 18:18:47 f1-oss1d8 kernel: [2610935.450237] LustreError: 1167:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 14 previous similar messages Oct 14 18:18:51 f1-oss1d8 kernel: [2610939.665431] LustreError: 76469:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 18:18:51 f1-oss1d8 kernel: [2610939.711746] LustreError: 76469:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 18:29:21 f1-oss1d8 kernel: [2611569.616260] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 18:29:21 f1-oss1d8 kernel: [2611569.638962] Lustre: Skipped 6 previous similar messages Oct 14 18:29:21 f1-oss1d8 kernel: [2611569.658420] LustreError: 90015:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 18:29:21 f1-oss1d8 kernel: [2611569.698965] LustreError: 90015:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 7 previous similar messages Oct 14 18:29:25 f1-oss1d8 kernel: [2611573.932101] LustreError: 79964:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 18:29:25 f1-oss1d8 kernel: [2611573.981083] LustreError: 79964:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 18:31:04 f1-oss1b4 kernel: [2611651.640395] LustreError: 94611:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 18:39:07 f1-oss1b6 kernel: [2612160.941006] LustreError: 15723:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 18:39:55 f1-oss1d8 kernel: [2612203.844865] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 18:39:55 f1-oss1d8 kernel: [2612203.844875] LustreError: 99504:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 18:39:55 f1-oss1d8 kernel: [2612203.844878] LustreError: 99504:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 7 previous similar messages Oct 14 18:39:55 f1-oss1d8 kernel: [2612203.937705] Lustre: Skipped 3 previous similar messages Oct 14 18:39:59 f1-oss1d8 kernel: [2612208.130230] LustreError: 83477:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 18:39:59 f1-oss1d8 kernel: [2612208.180481] LustreError: 83477:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 18:50:29 f1-oss1d8 kernel: [2612838.093217] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 18:50:29 f1-oss1d8 kernel: [2612838.115216] LustreError: 59349:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 18:50:29 f1-oss1d8 kernel: [2612838.115219] LustreError: 59349:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 18:50:29 f1-oss1d8 kernel: [2612838.187295] Lustre: Skipped 1 previous similar message Oct 14 18:50:33 f1-oss1d8 kernel: [2612842.390419] LustreError: 87476:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 18:50:33 f1-oss1d8 kernel: [2612842.439688] LustreError: 87476:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:01:03 f1-oss1d8 kernel: [2613472.311851] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x3104b4c:0x0], rc = 0 [1] Oct 14 19:01:03 f1-oss1d8 kernel: [2613472.311860] LustreError: 1166:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e9:0x0]: -115 Oct 14 19:01:03 f1-oss1d8 kernel: [2613472.311863] LustreError: 1166:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 19:01:03 f1-oss1d8 kernel: [2613472.406261] Lustre: Skipped 3 previous similar messages Oct 14 19:01:07 f1-oss1d8 kernel: [2613476.605051] LustreError: 91019:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:01:07 f1-oss1d8 kernel: [2613476.649393] LustreError: 91019:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:11:24 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 19:11:37 f1-oss1d8 kernel: [2614106.570227] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 19:11:37 f1-oss1d8 kernel: [2614106.579276] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 19:11:37 f1-oss1d8 kernel: [2614106.579280] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 19:11:37 f1-oss1d8 kernel: [2614106.666196] Lustre: Skipped 1 previous similar message Oct 14 19:11:41 f1-oss1d8 kernel: [2614110.871953] LustreError: 94517:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:11:41 f1-oss1d8 kernel: [2614110.918863] LustreError: 94517:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:15:58 f1-oss1e7 kernel: [2601589.485202] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 1282@gni1 ns: filter-f1-OST0026_UUID lock: ffff8801961a20c0/0xac020f1be2bade5e lrc: 3/0,0 mode: PW/PW res: [0x502a1c0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000000000020 nid: 1282@gni1 remote: 0xf384804fc9244713 expref: 6 pid: 22916 timeout: 6895287853 lvb_type: 0 Oct 14 19:19:31 f1-oss1d6 kernel: [2614594.334178] LustreError: 51941:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 19:22:11 f1-oss1d8 kernel: [2614740.789784] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 19:22:11 f1-oss1d8 kernel: [2614740.814934] Lustre: Skipped 1 previous similar message Oct 14 19:22:11 f1-oss1d8 kernel: [2614740.826015] LustreError: 90014:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 19:22:11 f1-oss1d8 kernel: [2614740.866426] LustreError: 90014:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 19:22:15 f1-oss1d8 kernel: [2614745.096232] LustreError: 98022:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:22:15 f1-oss1d8 kernel: [2614745.147896] LustreError: 98022:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:27:28 f1-oss1a8 kernel: [2615062.277977] LustreError: 118313:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 19:28:20 f1-oss1b8 kernel: [2615115.025719] LustreError: 26129:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 19:32:45 f1-oss1d8 kernel: [2615375.027214] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 19:32:45 f1-oss1d8 kernel: [2615375.032137] LustreError: 1164:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e8:0x0]: -115 Oct 14 19:32:45 f1-oss1d8 kernel: [2615375.032140] LustreError: 1164:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 8 previous similar messages Oct 14 19:32:45 f1-oss1d8 kernel: [2615375.124877] Lustre: Skipped 6 previous similar messages Oct 14 19:32:49 f1-oss1d8 kernel: [2615379.322312] LustreError: 101940:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:32:49 f1-oss1d8 kernel: [2615379.367582] LustreError: 101940:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:40:56 f1-oss1a8 kernel: [2615870.656593] LustreError: 116976:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 19:43:19 f1-oss1d8 kernel: [2616009.255765] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 19:43:19 f1-oss1d8 kernel: [2616009.255775] LustreError: 16192:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 19:43:19 f1-oss1d8 kernel: [2616009.255778] LustreError: 16192:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 19:43:19 f1-oss1d8 kernel: [2616009.354207] Lustre: Skipped 4 previous similar messages Oct 14 19:43:23 f1-oss1d8 kernel: [2616013.529040] LustreError: 105441:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:43:23 f1-oss1d8 kernel: [2616013.576639] LustreError: 105441:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 19:53:53 f1-oss1d8 kernel: [2616643.500238] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 19:53:53 f1-oss1d8 kernel: [2616643.523189] Lustre: Skipped 4 previous similar messages Oct 14 19:53:53 f1-oss1d8 kernel: [2616643.533984] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e3:0x0]: -115 Oct 14 19:53:53 f1-oss1d8 kernel: [2616643.574628] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 15 previous similar messages Oct 14 19:53:57 f1-oss1d8 kernel: [2616648.128507] LustreError: 108940:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 19:53:57 f1-oss1d8 kernel: [2616648.176121] LustreError: 108940:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:04:09 f1-oss1b1 kernel: [2607163.179340] LustreError: 33417:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 20:04:27 f1-oss1d8 kernel: [2617277.722727] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 20:04:27 f1-oss1d8 kernel: [2617277.752513] Lustre: Skipped 1 previous similar message Oct 14 20:04:27 f1-oss1d8 kernel: [2617277.763457] LustreError: 49921:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 20:04:27 f1-oss1d8 kernel: [2617277.803938] LustreError: 49921:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 6 previous similar messages Oct 14 20:04:31 f1-oss1d8 kernel: [2617282.027595] LustreError: 112515:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:04:31 f1-oss1d8 kernel: [2617282.075436] LustreError: 112515:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:06:29 f1-oss1d7 kernel: [2617391.144916] LustreError: 118310:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 20:11:22 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 20:15:01 f1-oss1d8 kernel: [2617911.953180] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 20:15:01 f1-oss1d8 kernel: [2617911.960116] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e8:0x0]: -115 Oct 14 20:15:01 f1-oss1d8 kernel: [2617911.960119] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 20:15:01 f1-oss1d8 kernel: [2617912.062489] Lustre: Skipped 5 previous similar messages Oct 14 20:15:05 f1-oss1d8 kernel: [2617916.609728] LustreError: 116016:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:15:05 f1-oss1d8 kernel: [2617916.654994] LustreError: 116016:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:24:25 f1-oss1d5 kernel: [2618464.434465] LustreError: 32144:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 20:25:35 f1-oss1d8 kernel: [2618546.267702] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 20:25:35 f1-oss1d8 kernel: [2618546.267939] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e8:0x0]: -115 Oct 14 20:25:35 f1-oss1d8 kernel: [2618546.267942] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 20:25:35 f1-oss1d8 kernel: [2618546.361816] Lustre: Skipped 1 previous similar message Oct 14 20:25:39 f1-oss1d8 kernel: [2618550.490956] LustreError: 120121:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:25:39 f1-oss1d8 kernel: [2618550.543241] LustreError: 120121:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:36:09 f1-oss1d8 kernel: [2619180.449288] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 20:36:09 f1-oss1d8 kernel: [2619180.475221] LustreError: 74121:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 20:36:09 f1-oss1d8 kernel: [2619180.475224] LustreError: 74121:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 9 previous similar messages Oct 14 20:36:09 f1-oss1d8 kernel: [2619180.550629] Lustre: Skipped 1 previous similar message Oct 14 20:36:13 f1-oss1d8 kernel: [2619184.754733] LustreError: 1069:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:36:13 f1-oss1d8 kernel: [2619184.803184] LustreError: 1069:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:46:43 f1-oss1d8 kernel: [2619814.658842] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 20:46:43 f1-oss1d8 kernel: [2619814.690101] Lustre: Skipped 3 previous similar messages Oct 14 20:46:43 f1-oss1d8 kernel: [2619814.709548] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 20:46:43 f1-oss1d8 kernel: [2619814.749822] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 20:46:47 f1-oss1d8 kernel: [2619818.953245] LustreError: 4708:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:46:47 f1-oss1d8 kernel: [2619819.002705] LustreError: 4708:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 20:57:17 f1-oss1d8 kernel: [2620448.938263] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 20:57:17 f1-oss1d8 kernel: [2620448.960106] LustreError: 50700:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 20:57:17 f1-oss1d8 kernel: [2620448.960109] LustreError: 50700:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 9 previous similar messages Oct 14 20:57:17 f1-oss1d8 kernel: [2620449.039500] Lustre: Skipped 1 previous similar message Oct 14 20:57:21 f1-oss1d8 kernel: [2620453.222249] LustreError: 8221:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 20:57:21 f1-oss1d8 kernel: [2620453.272067] LustreError: 8221:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 21:07:51 f1-oss1d8 kernel: [2621083.128796] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 21:07:51 f1-oss1d8 kernel: [2621083.143849] LustreError: 90015:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 21:07:51 f1-oss1d8 kernel: [2621083.143853] LustreError: 90015:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 21:07:51 f1-oss1d8 kernel: [2621083.228878] Lustre: Skipped 1 previous similar message Oct 14 21:07:55 f1-oss1d8 kernel: [2621087.413176] LustreError: 12197:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 21:07:55 f1-oss1d8 kernel: [2621087.461271] LustreError: 12197:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 21:11:24 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 21:12:44 f1-oss1b4 kernel: [2621355.353519] sd 6:0:0:25: Parameters changed Oct 14 21:12:44 f1-oss1b4 kernel: [2621355.750561] sd 6:0:0:85: Parameters changed Oct 14 21:12:44 f1-oss1b4 kernel: [2621355.759377] sd 7:0:0:85: Parameters changed Oct 14 21:12:46 f1-oss1b8 kernel: [2621381.639551] sd 7:0:0:25: Parameters changed Oct 14 21:12:49 f1-oss1b8 kernel: [2621383.651995] sd 6:0:0:85: Parameters changed Oct 14 21:12:54 f1-oss1b8 kernel: [2621386.657005] sd 6:0:0:25: Parameters changed Oct 14 21:18:25 f1-oss1d8 kernel: [2621717.371235] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 21:18:25 f1-oss1d8 kernel: [2621717.386195] LustreError: 121536:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e9:0x0]: -115 Oct 14 21:18:25 f1-oss1d8 kernel: [2621717.386198] LustreError: 121536:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 21:18:25 f1-oss1d8 kernel: [2621717.468421] Lustre: Skipped 4 previous similar messages Oct 14 21:18:29 f1-oss1d8 kernel: [2621721.697916] LustreError: 15783:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 21:18:29 f1-oss1d8 kernel: [2621721.750031] LustreError: 15783:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 21:21:04 f1-oss1d2 kernel: [2621876.383181] LustreError: 59973:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 21:28:59 f1-oss1d8 kernel: [2622351.599686] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 21:28:59 f1-oss1d8 kernel: [2622351.606697] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 21:28:59 f1-oss1d8 kernel: [2622351.606701] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 9 previous similar messages Oct 14 21:28:59 f1-oss1d8 kernel: [2622351.697567] Lustre: Skipped 1 previous similar message Oct 14 21:29:03 f1-oss1d8 kernel: [2622355.908644] LustreError: 19363:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 21:29:03 f1-oss1d8 kernel: [2622355.959988] LustreError: 19363:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 21:37:19 f1-oss1a7 kernel: [2622852.483916] LustreError: 9700:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 21:39:33 f1-oss1d8 kernel: [2622985.858288] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 21:39:33 f1-oss1d8 kernel: [2622985.886541] Lustre: Skipped 1 previous similar message Oct 14 21:39:33 f1-oss1d8 kernel: [2622985.897583] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 21:39:33 f1-oss1d8 kernel: [2622985.938170] LustreError: 59347:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 21:39:37 f1-oss1d8 kernel: [2622990.166841] LustreError: 22855:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 21:39:37 f1-oss1d8 kernel: [2622990.218767] LustreError: 22855:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 21:50:07 f1-oss1d8 kernel: [2623620.065760] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 21:50:07 f1-oss1d8 kernel: [2623620.089716] LustreError: 59358:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 21:50:07 f1-oss1d8 kernel: [2623620.089719] LustreError: 59358:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 21:50:07 f1-oss1d8 kernel: [2623620.166644] Lustre: Skipped 1 previous similar message Oct 14 21:50:11 f1-oss1d8 kernel: [2623624.354633] LustreError: 26368:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 21:50:11 f1-oss1d8 kernel: [2623624.398816] LustreError: 26368:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:00:41 f1-oss1d8 kernel: [2624254.309145] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x3104b4c:0x0], rc = 0 [1] Oct 14 22:00:41 f1-oss1d8 kernel: [2624254.309155] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e8:0x0]: -115 Oct 14 22:00:41 f1-oss1d8 kernel: [2624254.309159] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 22:00:41 f1-oss1d8 kernel: [2624254.406141] Lustre: Skipped 3 previous similar messages Oct 14 22:00:45 f1-oss1d8 kernel: [2624258.629417] LustreError: 30368:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:00:45 f1-oss1d8 kernel: [2624258.678208] LustreError: 30368:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:11:15 f1-oss1d8 kernel: [2624888.534810] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 22:11:15 f1-oss1d8 kernel: [2624888.539760] LustreError: 121234:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 22:11:15 f1-oss1d8 kernel: [2624888.539763] LustreError: 121234:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 22:11:15 f1-oss1d8 kernel: [2624888.635056] Lustre: Skipped 6 previous similar messages Oct 14 22:11:19 f1-oss1d8 kernel: [2624892.848015] LustreError: 33914:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:11:19 f1-oss1d8 kernel: [2624892.897688] LustreError: 33914:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:11:22 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 22:19:40 f1-oss1d4 kernel: [2625387.067591] LustreError: 118402:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 22:21:49 f1-oss1d8 kernel: [2625522.773528] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 22:21:49 f1-oss1d8 kernel: [2625522.773539] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 22:21:49 f1-oss1d8 kernel: [2625522.773542] LustreError: 99506:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 22:21:49 f1-oss1d8 kernel: [2625522.874216] Lustre: Skipped 3 previous similar messages Oct 14 22:21:53 f1-oss1d8 kernel: [2625527.081475] LustreError: 37422:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:21:53 f1-oss1d8 kernel: [2625527.126900] LustreError: 37422:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:22:15 f1-oss1b2 kernel: [2625558.399007] LustreError: 30797:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 22:22:15 f1-oss1b1 kernel: [2615452.092407] LustreError: 95187:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 22:22:15 f1-oss1a5 kernel: [2625549.892605] LustreError: 32140:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 22:32:23 f1-oss1d8 kernel: [2626157.051908] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 14 22:32:23 f1-oss1d8 kernel: [2626157.051955] LustreError: 1166:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 14 22:32:23 f1-oss1d8 kernel: [2626157.051958] LustreError: 1166:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 22:32:23 f1-oss1d8 kernel: [2626157.153382] Lustre: Skipped 3 previous similar messages Oct 14 22:32:27 f1-oss1d8 kernel: [2626161.340707] LustreError: 41243:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:32:27 f1-oss1d8 kernel: [2626161.386356] LustreError: 41243:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:42:57 f1-oss1d8 kernel: [2626791.317505] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 22:42:57 f1-oss1d8 kernel: [2626791.331401] LustreError: 59358:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 22:42:57 f1-oss1d8 kernel: [2626791.331405] LustreError: 59358:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 22:42:57 f1-oss1d8 kernel: [2626791.413313] Lustre: Skipped 1 previous similar message Oct 14 22:43:01 f1-oss1d8 kernel: [2626795.597026] LustreError: 44728:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:43:01 f1-oss1d8 kernel: [2626795.645666] LustreError: 44728:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 22:53:31 f1-oss1d8 kernel: [2627425.472169] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 22:53:31 f1-oss1d8 kernel: [2627425.472179] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 22:53:31 f1-oss1d8 kernel: [2627425.472182] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 22:53:31 f1-oss1d8 kernel: [2627425.582784] Lustre: Skipped 6 previous similar messages Oct 14 22:53:35 f1-oss1d8 kernel: [2627430.095447] LustreError: 48223:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 22:53:35 f1-oss1d8 kernel: [2627430.145127] LustreError: 48223:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:00:44 f1-oss1b8 kernel: [2621391.665352] sd 7:0:0:85: Parameters changed Oct 14 23:00:44 f1-oss1b8 kernel: [2627864.308463] LustreError: 110458:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 23:04:05 f1-oss1d8 kernel: [2628059.717581] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 23:04:05 f1-oss1d8 kernel: [2628059.741421] Lustre: Skipped 2 previous similar messages Oct 14 23:04:05 f1-oss1d8 kernel: [2628059.752355] LustreError: 50707:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 23:04:05 f1-oss1d8 kernel: [2628059.792550] LustreError: 50707:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 9 previous similar messages Oct 14 23:04:09 f1-oss1d8 kernel: [2628064.005203] LustreError: 51787:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:04:09 f1-oss1d8 kernel: [2628064.054135] LustreError: 51787:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:11:24 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 14 23:14:39 f1-oss1d8 kernel: [2628693.953994] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 23:14:39 f1-oss1d8 kernel: [2628693.980930] Lustre: Skipped 6 previous similar messages Oct 14 23:14:39 f1-oss1d8 kernel: [2628693.991955] LustreError: 119334:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e3:0x0]: -115 Oct 14 23:14:39 f1-oss1d8 kernel: [2628694.040964] LustreError: 119334:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 9 previous similar messages Oct 14 23:14:43 f1-oss1d8 kernel: [2628698.271948] LustreError: 55273:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:14:43 f1-oss1d8 kernel: [2628698.323514] LustreError: 55273:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:25:13 f1-oss1d8 kernel: [2629328.198482] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 14 23:25:13 f1-oss1d8 kernel: [2629328.209767] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 23:25:13 f1-oss1d8 kernel: [2629328.209770] LustreError: 59334:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 7 previous similar messages Oct 14 23:25:13 f1-oss1d8 kernel: [2629328.291534] Lustre: Skipped 1 previous similar message Oct 14 23:25:17 f1-oss1d8 kernel: [2629332.790829] LustreError: 58776:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:25:17 f1-oss1d8 kernel: [2629332.843188] LustreError: 58776:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:32:17 f1-oss1a3 kernel: [2629771.290117] LustreError: 29400:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 23:32:37 f1-oss1b7 kernel: [2629782.601539] LustreError: 25743:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 23:35:47 f1-oss1d8 kernel: [2629962.415203] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x3104b4c:0x0], rc = 0 [1] Oct 14 23:35:47 f1-oss1d8 kernel: [2629962.415212] LustreError: 49921:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e8:0x0]: -115 Oct 14 23:35:47 f1-oss1d8 kernel: [2629962.415215] LustreError: 49921:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 14 23:35:47 f1-oss1d8 kernel: [2629962.510213] Lustre: Skipped 3 previous similar messages Oct 14 23:35:51 f1-oss1d8 kernel: [2629966.706349] LustreError: 63181:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:35:51 f1-oss1d8 kernel: [2629966.752538] LustreError: 63181:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:46:21 f1-oss1d8 kernel: [2630596.650639] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x3104b4c:0x0], rc = 0 [1] Oct 14 23:46:21 f1-oss1d8 kernel: [2630596.650670] LustreError: 62212:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e9:0x0]: -115 Oct 14 23:46:21 f1-oss1d8 kernel: [2630596.650674] LustreError: 62212:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 10 previous similar messages Oct 14 23:46:21 f1-oss1d8 kernel: [2630596.749076] Lustre: Skipped 3 previous similar messages Oct 14 23:46:25 f1-oss1d8 kernel: [2630600.910007] LustreError: 66674:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:46:25 f1-oss1d8 kernel: [2630600.961752] LustreError: 66674:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 14 23:56:55 f1-oss1d8 kernel: [2631230.897033] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 14 23:56:55 f1-oss1d8 kernel: [2631230.928307] Lustre: Skipped 1 previous similar message Oct 14 23:56:55 f1-oss1d8 kernel: [2631230.939100] LustreError: 50702:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 14 23:56:55 f1-oss1d8 kernel: [2631230.979705] LustreError: 50702:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 15 previous similar messages Oct 14 23:56:59 f1-oss1d8 kernel: [2631235.494558] LustreError: 70195:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 23:56:59 f1-oss1d8 kernel: [2631235.541331] LustreError: 70195:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 15 00:07:29 f1-oss1d8 kernel: [2631865.114623] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e3:0x0], rc = 0 [1] Oct 15 00:07:29 f1-oss1d8 kernel: [2631865.120609] LustreError: 121234:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e4:0x0]: -115 Oct 15 00:07:29 f1-oss1d8 kernel: [2631865.120612] LustreError: 121234:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 7 previous similar messages Oct 15 00:07:29 f1-oss1d8 kernel: [2631865.208317] Lustre: Skipped 1 previous similar message Oct 15 00:07:33 f1-oss1d8 kernel: [2631869.384275] LustreError: 74083:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 15 00:07:33 f1-oss1d8 kernel: [2631869.430691] LustreError: 74083:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 15 00:11:23 lustre-mgmt02.ncrc.gov kernel: ACPI: No handler for Region [POWR] (ffff880834fc72b8) [IPMI] Oct 15 00:18:03 f1-oss1d8 kernel: [2632499.405124] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 15 00:18:03 f1-oss1d8 kernel: [2632499.415075] LustreError: 59387:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 15 00:18:03 f1-oss1d8 kernel: [2632499.415078] LustreError: 59387:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 12 previous similar messages Oct 15 00:18:03 f1-oss1d8 kernel: [2632499.498156] Lustre: Skipped 1 previous similar message Oct 15 00:18:07 f1-oss1d8 kernel: [2632503.707428] LustreError: 77577:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 15 00:18:07 f1-oss1d8 kernel: [2632503.760000] LustreError: 77577:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 15 00:28:37 f1-oss1d8 kernel: [2633133.582813] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e4:0x0], rc = 0 [1] Oct 15 00:28:37 f1-oss1d8 kernel: [2633133.582823] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e3:0x0]: -115 Oct 15 00:28:37 f1-oss1d8 kernel: [2633133.582826] LustreError: 90012:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 15 00:28:37 f1-oss1d8 kernel: [2633133.676649] Lustre: Skipped 5 previous similar messages Oct 15 00:28:41 f1-oss1d8 kernel: [2633137.892407] LustreError: 81070:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 15 00:28:41 f1-oss1d8 kernel: [2633137.939258] LustreError: 81070:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 15 00:39:11 f1-oss1d8 kernel: [2633767.819787] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e8:0x0], rc = 0 [1] Oct 15 00:39:11 f1-oss1d8 kernel: [2633767.819798] LustreError: 121531:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x3104b4c:0x0]: -115 Oct 15 00:39:11 f1-oss1d8 kernel: [2633767.819801] LustreError: 121531:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 15 00:39:11 f1-oss1d8 kernel: [2633767.916197] Lustre: Skipped 6 previous similar messages Oct 15 00:39:15 f1-oss1d8 kernel: [2633772.116183] LustreError: 84577:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 15 00:39:15 f1-oss1d8 kernel: [2633772.168522] LustreError: 84577:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages Oct 15 00:40:00 f1-oss1a3 kernel: [2633835.375741] LustreError: 85845:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 15 00:40:00 f1-oss1a5 kernel: [2633818.056874] LustreError: 95085:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 15 00:46:18 f1-oss1c5 kernel: [2621413.996153] INFO: task ll_ost_io00_001:16655 blocked for more than 120 seconds. Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.011446] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.031025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.060874] ll_ost_io00_0 D 0000000000000001 0 16655 2 0x00000000 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.081091] ffff881021f6d720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.102162] ffff881021f6d6e0 ffffffffa029961c ffff880028047e68 ffff881021f6d7c0 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.131550] 0000000000000002 ffffffff81124920 ffff8810226f65f8 ffff881021f6dfd8 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.152617] Call Trace: Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.161478] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.190579] [] ? sync_page+0x0/0x50 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.202009] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.221848] [] ? sync_page+0x0/0x50 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.241492] [] io_schedule+0x73/0xc0 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.260991] [] sync_page+0x3d/0x50 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.280620] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.300627] [] __lock_page+0x67/0x70 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.311869] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:18 f1-oss1c5 kernel: [2621414.332032] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.351727] [] find_lock_page+0x50/0x80 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.371686] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.391690] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.420470] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.441049] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.461251] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.481327] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.510872] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.521962] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.550542] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.571132] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.591634] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.621132] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.642008] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.661877] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.690581] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.711953] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.741119] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.761172] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.781537] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.802028] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.822067] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.850860] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.871023] [] kthread+0x9e/0xc0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.882260] [] child_rip+0xa/0x20 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.901758] [] ? kthread+0x0/0xc0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.921258] [] ? child_rip+0x0/0x20 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.941073] INFO: task ll_ost_io01_002:16659 blocked for more than 120 seconds. Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.962382] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:19 f1-oss1c5 kernel: [2621414.981951] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.012072] ll_ost_io01_0 D 0000000000000007 0 16659 2 0x00000000 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.032354] ffff881021f7f720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.062173] ffff881021f7f6e0 ffffffffa029961c ffff881021f7f690 ffffffff81014a29 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.090715] ffff881021f7f6d0 ffffffff810aab71 ffff881021f7dad8 ffff881021f7ffd8 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.112192] Call Trace: Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.121313] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.141956] [] ? read_tsc+0x9/0x20 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.161686] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.181536] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.201512] [] ? sync_page+0x0/0x50 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.221033] [] io_schedule+0x73/0xc0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.232368] [] sync_page+0x3d/0x50 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.251866] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.271950] [] __lock_page+0x67/0x70 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.291727] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.311788] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.331566] [] find_lock_page+0x50/0x80 Oct 15 00:46:19 f1-oss1c5 kernel: [2621415.351381] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.371391] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.391798] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.420824] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.441036] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.461049] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.482028] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.501787] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.521805] [] ? libcfs_debug_vmsg2+0x5d3/0xbd0 [libcfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.542508] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.571856] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.592765] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.620901] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.641530] [] ? ptlrpc_update_export_timer+0x1ae/0x560 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.662907] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.692420] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.712624] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.741206] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.761793] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.781948] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.802265] [] kthread+0x9e/0xc0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.821722] [] child_rip+0xa/0x20 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.841394] [] ? kthread+0x0/0xc0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.861014] [] ? child_rip+0x0/0x20 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.872274] INFO: task ll_ost_io02_001:16661 blocked for more than 120 seconds. Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.901577] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.921610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.951418] ll_ost_io02_0 D 0000000000000008 0 16661 2 0x00000000 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.971606] ffff881021fa9720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:20 f1-oss1c5 kernel: [2621415.993113] ffff881021fa96e0 ffffffffa029961c ffff880028047e68 ffff881021fa97c0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.022186] 0000000000000002 ffffffff81124920 ffff881021f7c5f8 ffff881021fa9fd8 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.042940] Call Trace: Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.052413] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.081291] [] ? sync_page+0x0/0x50 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.092453] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.112453] [] ? sync_page+0x0/0x50 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.132094] [] io_schedule+0x73/0xc0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.151741] [] sync_page+0x3d/0x50 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.171128] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.182803] [] __lock_page+0x67/0x70 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.202420] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.222553] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.242390] [] find_lock_page+0x50/0x80 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.262280] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.282424] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:20 f1-oss1c5 kernel: [2621416.302781] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.331777] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.361974] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.381232] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.402328] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.422016] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.441973] [] ? libcfs_debug_vmsg2+0x5d3/0xbd0 [libcfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.462835] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.492202] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.512899] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.532689] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.561260] [] ? __switch_to+0x7d/0x320 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.572978] [] ? ptlrpc_update_export_timer+0x1ae/0x560 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.602770] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.632046] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.652060] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.672647] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.701501] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.721539] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.741896] [] kthread+0x9e/0xc0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.761321] [] child_rip+0xa/0x20 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.772636] [] ? kthread+0x0/0xc0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.792082] [] ? child_rip+0x0/0x20 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.811747] INFO: task ll_ost_io01_008:76094 blocked for more than 120 seconds. Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.833190] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.852658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.882690] ll_ost_io01_0 D 0000000000000004 0 76094 2 0x00000000 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.911362] ffff880fffa37720 0000000000000046 ffff880fffa376e8 ffff880fffa376e4 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.932695] ffff880fffa376e0 ffff88087fe71700 0009500853463afa ffff8800446f5900 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.961385] 0000000000000400 000000019c2a0970 ffff880fff7bfad8 ffff880fffa37fd8 Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.982889] Call Trace: Oct 15 00:46:21 f1-oss1c5 kernel: [2621416.992161] [] ? sync_page+0x0/0x50 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.011598] [] io_schedule+0x73/0xc0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.022769] [] sync_page+0x3d/0x50 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.042626] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.062578] [] __lock_page+0x67/0x70 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.082386] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.102234] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.122084] [] find_lock_page+0x50/0x80 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.141893] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.161931] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.182418] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.202920] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.231505] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.251558] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.272650] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.292507] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:21 f1-oss1c5 kernel: [2621417.312470] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.332999] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.361784] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.382874] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.412007] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.431614] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.452371] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.482003] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.502961] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.522900] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.551677] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.572038] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.592232] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.612262] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.632567] [] kthread+0x9e/0xc0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.652044] [] child_rip+0xa/0x20 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.663321] [] ? kthread+0x0/0xc0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.682609] [] ? child_rip+0x0/0x20 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.702462] INFO: task ll_ost_io01_012:76361 blocked for more than 120 seconds. Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.732053] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.751929] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.773527] ll_ost_io01_0 D 0000000000000007 0 76361 2 0x00000000 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.802219] ffff880ffee11720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.823540] ffff880ffee116e0 ffffffffa029961c ffff880028047e68 ffff880ffee117c0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.852680] 0000000000000002 ffffffff81124920 ffff880ffff0a5f8 ffff880ffee11fd8 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.873566] Call Trace: Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.882688] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.903372] [] ? sync_page+0x0/0x50 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.922908] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.942986] [] ? sync_page+0x0/0x50 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.962541] [] io_schedule+0x73/0xc0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.981824] [] sync_page+0x3d/0x50 Oct 15 00:46:22 f1-oss1c5 kernel: [2621417.993209] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.013467] [] __lock_page+0x67/0x70 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.032998] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.052964] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.072934] [] find_lock_page+0x50/0x80 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.092618] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.112842] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.133206] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.161890] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.182282] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.202065] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.223380] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.242894] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.263029] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.291926] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:22 f1-oss1c5 kernel: [2621418.312307] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.333429] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.362688] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.382506] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.402913] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.432585] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.453606] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.482005] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.502378] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.523223] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.542977] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.563465] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.583504] [] kthread+0x9e/0xc0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.603113] [] child_rip+0xa/0x20 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.622761] [] ? kthread+0x0/0xc0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.642219] [] ? child_rip+0x0/0x20 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.653447] INFO: task ll_ost_io00_009:107949 blocked for more than 120 seconds. Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.683176] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.702798] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.732453] ll_ost_io00_0 D 0000000000000002 0 107949 2 0x00000000 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.752799] ffff880ffec1d720 0000000000000046 0000000000000000 ffff880ffec1d6e4 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.782084] ffff880f00000000 ffff88087fe71300 00095008534698f9 ffff8800446b5900 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.803709] 0000000000000400 000000019c2a0974 ffff880fff38fad8 ffff880ffec1dfd8 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.832677] Call Trace: Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.834481] [] ? sync_page+0x0/0x50 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.853222] [] io_schedule+0x73/0xc0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.873033] [] sync_page+0x3d/0x50 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.892372] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.912304] [] __lock_page+0x67/0x70 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.923724] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.952149] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.963669] [] find_lock_page+0x50/0x80 Oct 15 00:46:23 f1-oss1c5 kernel: [2621418.983285] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.003452] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.032241] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.052783] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.072966] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.093021] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.114005] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.133747] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.153813] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.182501] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.203034] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.223970] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.253263] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.272906] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.293461] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.323291] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:23 f1-oss1c5 kernel: [2621419.344115] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.372489] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.393118] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.413622] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.433834] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.472786] [] kthread+0x9e/0xc0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.483333] [] child_rip+0xa/0x20 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.502724] [] ? kthread+0x0/0xc0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.522538] [] ? child_rip+0x0/0x20 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.533801] INFO: task ll_ost_io03_006:107951 blocked for more than 120 seconds. Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.563533] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.583167] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.612783] ll_ost_io03_0 D 000000000000000c 0 107951 2 0x00000000 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.632986] ffff880ffd577720 0000000000000046 0000000000000000 ffff880838f62f00 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.654517] ffff880ffd5776e0 ffffffffa029961c 0009500853470073 ffff880ffd5777c0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.683696] 0000000000000002 000000019c2a0993 ffff880ffe9905f8 ffff880ffd577fd8 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.713202] Call Trace: Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.722452] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.743025] [] ? sync_page+0x0/0x50 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.762860] [] io_schedule+0x73/0xc0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.782438] [] sync_page+0x3d/0x50 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.793723] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.813623] [] __lock_page+0x67/0x70 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.833448] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.853241] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.873070] [] find_lock_page+0x50/0x80 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.892874] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.912713] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.933080] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.953637] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.974040] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:24 f1-oss1c5 kernel: [2621419.994144] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.023493] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.043149] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.063347] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.083861] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.112612] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.133537] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.154338] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.174208] [] ? common_interrupt+0xe/0x13 Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.202669] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.224155] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.253606] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.273496] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.293988] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:24 f1-oss1c5 kernel: [2621420.322713] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.342727] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.362938] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.383247] [] kthread+0x9e/0xc0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.402707] [] child_rip+0xa/0x20 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.413843] [] ? kthread+0x0/0xc0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.433426] [] ? child_rip+0x0/0x20 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.452906] INFO: task ll_ost_io02_017:107967 blocked for more than 120 seconds. Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.474022] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.493836] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.523670] ll_ost_io02_0 D 0000000000000008 0 107967 2 0x00000000 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.544013] ffff880ffe63f720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.573714] ffff880ffe63f6e0 ffffffffa029961c ffff880028047e68 ffff880ffe63f7c0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.594585] 0000000000000002 ffffffff81124920 ffff880ffedce5f8 ffff880ffe63ffd8 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.623688] Call Trace: Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.632968] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.653558] [] ? sync_page+0x0/0x50 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.673182] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.693160] [] ? sync_page+0x0/0x50 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.712835] [] io_schedule+0x73/0xc0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.724061] [] sync_page+0x3d/0x50 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.743596] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.763578] [] __lock_page+0x67/0x70 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.783222] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.803257] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.823055] [] find_lock_page+0x50/0x80 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.842881] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.854373] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.883245] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.903841] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.923995] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.944193] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.973459] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:25 f1-oss1c5 kernel: [2621420.993129] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.013370] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.033984] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.054342] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.083881] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.104655] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.124396] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.153559] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.174794] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.204252] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.224167] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.244565] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.273617] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.293664] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.314027] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:25 f1-oss1c5 kernel: [2621421.333956] [] kthread+0x9e/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.353410] [] child_rip+0xa/0x20 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.373090] [] ? kthread+0x0/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.384544] [] ? child_rip+0x0/0x20 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.404047] INFO: task ll_ost_io01_013:107972 blocked for more than 120 seconds. Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.433696] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.453484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.474821] ll_ost_io01_0 D 0000000000000006 0 107972 2 0x00000000 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.503478] ffff880ffd1cd720 0000000000000046 0000000000000000 ffff880838f62f00 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.524792] ffff880ffd1cd6e0 ffffffffa029961c 000950085348dea8 ffff880ffd1cd7c0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.553936] 0000000000000002 000000019c2a0970 ffff880fff65b068 ffff880ffd1cdfd8 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.574826] Call Trace: Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.584082] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.613124] [] ? sync_page+0x0/0x50 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.624433] [] io_schedule+0x73/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.644072] [] sync_page+0x3d/0x50 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.663731] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.683581] [] __lock_page+0x67/0x70 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.703158] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.723201] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.743166] [] find_lock_page+0x50/0x80 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.754619] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.774672] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.803596] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.824000] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.844575] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.864540] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.893808] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.913441] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.933890] [] ? libcfs_debug_vmsg2+0x5d3/0xbd0 [libcfs] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.954400] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621421.984079] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.004794] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.024661] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.053555] [] ? ptlrpc_update_export_timer+0x1ae/0x560 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.074653] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.104074] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.123943] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.144315] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.164955] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.193368] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.213607] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.233784] [] kthread+0x9e/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.253365] [] child_rip+0xa/0x20 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.264522] [] ? kthread+0x0/0xc0 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.284168] [] ? child_rip+0x0/0x20 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.303607] INFO: task ll_ost_io02_021:107986 blocked for more than 120 seconds. Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.325139] Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1 Oct 15 00:46:26 f1-oss1c5 kernel: [2621422.344881] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.374745] ll_ost_io02_0 D 0000000000000008 0 107986 2 0x00000000 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.395037] ffff880ffce1b720 0000000000000046 ffff880819c72bc0 ffff880838f62f00 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.424800] ffff880ffce1b6e0 ffffffffa029961c 0000000000000870 0000000000000008 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.454037] 0000000000000000 ffff8808723719e0 ffff880ffcd185f8 ffff880ffce1bfd8 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.474980] Call Trace: Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.484261] [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.504926] [] ? ktime_get_ts+0xb1/0xf0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.524579] [] ? sync_page+0x0/0x50 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.554694] [] io_schedule+0x73/0xc0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.564780] [] sync_page+0x3d/0x50 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.584596] [] __wait_on_bit_lock+0x5a/0xc0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.604650] [] __lock_page+0x67/0x70 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.624454] [] ? wake_bit_function+0x0/0x50 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.644304] [] ? find_get_page+0x1e/0xa0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.664161] [] find_lock_page+0x50/0x80 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.683925] [] find_or_create_page+0x3f/0xb0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.703960] [] osd_get_page+0x11a/0x180 [osd_ldiskfs] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.724514] [] osd_bufs_get+0x1f6/0x370 [osd_ldiskfs] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.744887] [] ofd_preprw_read+0x14b/0x7f0 [ofd] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.765010] [] ofd_preprw+0x77a/0x1490 [ofd] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.785034] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.814365] [] obd_preprw+0x121/0x390 [ost] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.834041] [] ost_brw_read+0xd1e/0x11d0 [ost] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.854234] [] ? libcfs_debug_vmsg2+0x5d3/0xbd0 [libcfs] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.874924] [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.904419] [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.925264] [] ost_handle+0x24a8/0x44f0 [ost] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.945077] [] ? ost_rw_hpreq_check+0x25b/0x500 [ost] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.973858] [] ? ptlrpc_update_export_timer+0x1ae/0x560 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621422.995334] [] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.024635] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.044644] [] ? lc_watchdog_touch+0x65/0x170 [libcfs] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.065201] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.093887] [] ? __wake_up_common+0x59/0x90 Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.113929] [] ptlrpc_main+0xaed/0x1930 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.134343] [] ? ptlrpc_main+0x0/0x1930 [ptlrpc] Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.154469] [] kthread+0x9e/0xc0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.174095] [] child_rip+0xa/0x20 Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.185306] [] ? kthread+0x0/0xc0 Oct 15 00:46:27 f1-oss1c5 kernel: [2621423.204891] [] ? child_rip+0x0/0x20