Jan 29 11:40:44 oak-gw04 kernel: Lustre: Mounted oak-client Jan 29 12:45:20 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258720/real 1517258720] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258732 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Jan 29 12:45:20 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:21 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:21 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258721/real 1517258721] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258733 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:45:21 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Jan 29 12:45:21 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:21 oak-gw04 kernel: Lustre: Skipped 42 previous similar messages Jan 29 12:45:21 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:21 oak-gw04 kernel: Lustre: Skipped 82 previous similar messages Jan 29 12:45:22 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258722/real 1517258722] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258734 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:45:22 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 145 previous similar messages Jan 29 12:45:22 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:22 oak-gw04 kernel: Lustre: Skipped 144 previous similar messages Jan 29 12:45:22 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:22 oak-gw04 kernel: Lustre: Skipped 133 previous similar messages Jan 29 12:45:24 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258724/real 1517258724] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258736 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:45:24 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 262 previous similar messages Jan 29 12:45:24 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:24 oak-gw04 kernel: Lustre: Skipped 262 previous similar messages Jan 29 12:45:24 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:24 oak-gw04 kernel: Lustre: Skipped 258 previous similar messages Jan 29 12:45:28 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258728/real 1517258728] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258740 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:45:28 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 450 previous similar messages Jan 29 12:45:28 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:28 oak-gw04 kernel: Lustre: Skipped 450 previous similar messages Jan 29 12:45:28 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:28 oak-gw04 kernel: Lustre: Skipped 449 previous similar messages Jan 29 12:45:36 oak-gw04 kernel: Lustre: 9089:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258736/real 1517258736] req@ffff880283315500 x1590957016747344/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 3440/1152 e 683 to 1 dl 1517258773 ref 3 fl Rpc:eX/2/ffffffff rc -11/-1 Jan 29 12:45:36 oak-gw04 kernel: Lustre: 9089:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 972 previous similar messages Jan 29 12:45:36 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:36 oak-gw04 kernel: Lustre: Skipped 884 previous similar messages Jan 29 12:45:36 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:36 oak-gw04 kernel: Lustre: Skipped 891 previous similar messages Jan 29 12:45:52 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258752/real 1517258752] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258764 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:45:52 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1458 previous similar messages Jan 29 12:45:52 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:45:52 oak-gw04 kernel: Lustre: Skipped 1382 previous similar messages Jan 29 12:45:52 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:45:52 oak-gw04 kernel: Lustre: Skipped 1368 previous similar messages Jan 29 12:46:24 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258784/real 1517258784] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258796 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:46:24 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2459 previous similar messages Jan 29 12:46:24 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:46:24 oak-gw04 kernel: Lustre: Skipped 2456 previous similar messages Jan 29 12:46:24 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:46:24 oak-gw04 kernel: Lustre: Skipped 2456 previous similar messages Jan 29 12:47:28 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258848/real 1517258848] req@ffff880283316700 x1590957016747328/t0(0) o4->oak-OST001c-osc-ffff88040df08000@10.0.2.101@o2ib5:6/4 lens 4176/1336 e 0 to 1 dl 1517258860 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Jan 29 12:47:28 oak-gw04 kernel: Lustre: 9090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5286 previous similar messages Jan 29 12:47:28 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:47:28 oak-gw04 kernel: Lustre: Skipped 5137 previous similar messages Jan 29 12:47:28 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 12:47:28 oak-gw04 kernel: Lustre: Skipped 5126 previous similar messages Jan 29 12:47:50 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:47:50 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:47:50 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:47:50 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:47:50 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:47:50 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:47:50 oak-gw04 kernel: Call Trace: Jan 29 12:47:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:47:50 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:47:50 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:47:50 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:47:50 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:47:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:47:50 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:47:50 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:47:50 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:47:50 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:47:50 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:47:50 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:47:50 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:47:50 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:47:50 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:47:50 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:47:50 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:47:50 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:47:50 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:47:50 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:47:50 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:47:50 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:47:50 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:49:50 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:49:50 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:49:50 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:49:50 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:49:50 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:49:50 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:49:50 oak-gw04 kernel: Call Trace: Jan 29 12:49:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:49:50 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:49:50 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:49:50 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:49:50 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:49:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:49:50 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:49:50 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:49:50 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:49:50 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:49:50 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:49:50 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:49:50 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:49:50 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:49:50 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:49:50 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:49:50 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:49:50 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:49:50 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:49:50 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:49:50 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:49:50 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:49:50 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:49:51 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 0 seconds Jan 29 12:49:51 oak-gw04 kernel: Lustre: 9087:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517258966/real 1517258991] req@ffff880409fb9b00 x1590957017613648/t0(0) o400->oak-OST0002-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1517259012 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 12:49:51 oak-gw04 kernel: Lustre: oak-OST0004-osc-ffff88040df08000: Connection to oak-OST0004 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 12:49:51 oak-gw04 kernel: Lustre: Skipped 6529 previous similar messages Jan 29 12:49:51 oak-gw04 kernel: Lustre: 9087:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6605 previous similar messages Jan 29 12:50:04 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 3 seconds Jan 29 12:50:04 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 6 previous similar messages Jan 29 12:50:16 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 25 seconds Jan 29 12:50:16 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Jan 29 12:50:29 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 28 seconds Jan 29 12:50:29 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Jan 29 12:50:41 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 50 seconds Jan 29 12:50:41 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Jan 29 12:50:54 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 53 seconds Jan 29 12:50:54 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Jan 29 12:51:19 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 78 seconds Jan 29 12:51:19 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 15 previous similar messages Jan 29 12:51:50 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:51:50 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:51:50 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:51:50 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:51:50 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:51:50 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:51:50 oak-gw04 kernel: Call Trace: Jan 29 12:51:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:51:50 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:51:50 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:51:50 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:51:50 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:51:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:51:50 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:51:50 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:51:50 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:51:50 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:51:50 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:51:50 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:51:50 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:51:50 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:51:50 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:51:50 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:51:50 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:51:50 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:51:50 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:51:50 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:51:50 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:51:50 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:51:50 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:51:57 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 5 seconds Jan 29 12:51:57 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 5 previous similar messages Jan 29 12:53:13 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 22 seconds Jan 29 12:53:13 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 18 previous similar messages Jan 29 12:53:50 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:53:50 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:53:50 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:53:50 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:53:50 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:53:50 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:53:50 oak-gw04 kernel: Call Trace: Jan 29 12:53:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:53:50 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:53:50 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:53:50 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:53:50 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:53:50 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:53:50 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:53:50 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:53:50 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:53:50 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:53:50 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:53:50 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:53:50 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:53:50 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:53:50 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:53:50 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:53:50 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:53:50 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:53:50 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:53:50 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:53:50 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:53:50 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:53:50 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:54:30 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517259270/real 1517259270] req@ffff8800589d2700 x1590957017628320/t0(0) o8->oak-OST0000-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517259291 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 12:54:30 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 127 previous similar messages Jan 29 12:55:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:55:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:55:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:55:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:55:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:55:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:55:51 oak-gw04 kernel: Call Trace: Jan 29 12:55:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:55:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:55:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:55:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:55:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:55:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:55:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:55:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:55:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:55:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:55:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:55:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:55:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:55:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:55:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:55:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:55:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:55:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:55:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:55:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:55:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:55:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:55:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:57:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:57:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:57:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:57:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:57:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:57:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:57:51 oak-gw04 kernel: Call Trace: Jan 29 12:57:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:57:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:57:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:57:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:57:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:57:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:57:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:57:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:57:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:57:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:57:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:57:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:57:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:57:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:57:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:57:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:57:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:57:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:57:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:57:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:57:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:57:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:57:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 12:59:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 12:59:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 12:59:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 12:59:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 12:59:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 12:59:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 12:59:51 oak-gw04 kernel: Call Trace: Jan 29 12:59:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:59:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 12:59:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 12:59:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 12:59:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 12:59:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 12:59:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 12:59:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 12:59:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 12:59:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 12:59:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 12:59:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 12:59:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 12:59:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 12:59:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 12:59:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 12:59:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 12:59:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 12:59:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 12:59:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 12:59:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 12:59:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 12:59:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 13:00:47 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 1 seconds Jan 29 13:00:47 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 20 previous similar messages Jan 29 13:01:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 13:01:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 13:01:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 13:01:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 13:01:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 13:01:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 13:01:51 oak-gw04 kernel: Call Trace: Jan 29 13:01:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:01:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 13:01:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 13:01:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 13:01:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 13:01:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:01:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 13:01:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 13:01:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 13:01:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 13:01:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 13:01:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 13:01:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 13:01:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 13:01:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 13:01:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 13:01:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 13:01:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 13:01:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 13:01:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 13:01:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 13:01:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 13:01:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 13:03:06 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517259745/real 1517259786] req@ffff88042edb3900 x1590957017652768/t0(0) o8->oak-OST0020-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517259801 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 13:03:06 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 221 previous similar messages Jan 29 13:03:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 13:03:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 13:03:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 13:03:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 13:03:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 13:03:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 13:03:51 oak-gw04 kernel: Call Trace: Jan 29 13:03:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:03:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 13:03:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 13:03:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 13:03:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 13:03:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:03:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 13:03:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 13:03:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 13:03:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 13:03:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 13:03:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 13:03:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 13:03:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 13:03:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 13:03:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 13:03:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 13:03:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 13:03:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 13:03:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 13:03:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 13:03:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 13:03:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 13:05:50 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 4 seconds Jan 29 13:05:50 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 82 previous similar messages Jan 29 13:05:51 oak-gw04 kernel: INFO: task smbd:11865 blocked for more than 120 seconds. Jan 29 13:05:51 oak-gw04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 29 13:05:51 oak-gw04 kernel: smbd D ffffffff816a7610 0 11865 1363 0x00000080 Jan 29 13:05:51 oak-gw04 kernel: ffff880425ef37f0 0000000000000086 ffff880417246eb0 ffff880425ef3fd8 Jan 29 13:05:51 oak-gw04 kernel: ffff880425ef3fd8 ffff880425ef3fd8 ffff880417246eb0 ffff88043fc16cc0 Jan 29 13:05:51 oak-gw04 kernel: 0000000000000000 7fffffffffffffff ffff88043ff412e8 ffffffff816a7610 Jan 29 13:05:51 oak-gw04 kernel: Call Trace: Jan 29 13:05:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:05:51 oak-gw04 kernel: [] schedule+0x29/0x70 Jan 29 13:05:51 oak-gw04 kernel: [] schedule_timeout+0x239/0x2c0 Jan 29 13:05:51 oak-gw04 kernel: [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jan 29 13:05:51 oak-gw04 kernel: [] ? ll_xattr_cache_refill+0xca5/0x1890 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] ? kvm_clock_get_cycles+0x1e/0x20 Jan 29 13:05:51 oak-gw04 kernel: [] ? bit_wait+0x50/0x50 Jan 29 13:05:51 oak-gw04 kernel: [] io_schedule_timeout+0xad/0x130 Jan 29 13:05:51 oak-gw04 kernel: [] io_schedule+0x18/0x20 Jan 29 13:05:51 oak-gw04 kernel: [] bit_wait_io+0x11/0x50 Jan 29 13:05:51 oak-gw04 kernel: [] __wait_on_bit+0x65/0x90 Jan 29 13:05:51 oak-gw04 kernel: [] ? ll_xattr_cache_get+0x14b/0x4e0 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] wait_on_page_bit+0x81/0xa0 Jan 29 13:05:51 oak-gw04 kernel: [] ? wake_bit_function+0x40/0x40 Jan 29 13:05:51 oak-gw04 kernel: [] vvp_page_assume+0x32/0xa0 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] cl_page_assume+0x68/0x240 [obdclass] Jan 29 13:05:51 oak-gw04 kernel: [] ll_write_begin+0x100/0x830 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Jan 29 13:05:51 oak-gw04 kernel: [] ? osc_lock_enqueue+0x2cb/0x9e0 [osc] Jan 29 13:05:51 oak-gw04 kernel: [] __generic_file_aio_write+0x1e2/0x400 Jan 29 13:05:51 oak-gw04 kernel: [] __generic_file_write_iter+0xcb/0x340 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] vvp_io_write_start+0x610/0x810 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] ? cl_lock_request+0x67/0x1f0 [obdclass] Jan 29 13:05:51 oak-gw04 kernel: [] cl_io_start+0x65/0x130 [obdclass] Jan 29 13:05:51 oak-gw04 kernel: [] cl_io_loop+0x12e/0xc90 [obdclass] Jan 29 13:05:51 oak-gw04 kernel: [] ll_file_io_generic+0x498/0xc40 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Jan 29 13:05:51 oak-gw04 kernel: [] vfs_write+0xbd/0x1e0 Jan 29 13:05:51 oak-gw04 kernel: [] SyS_pwrite64+0x92/0xc0 Jan 29 13:05:51 oak-gw04 kernel: [] system_call_fastpath+0x16/0x1b Jan 29 13:13:11 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1517260370/real 0] req@ffff88041830a400 x1590957017680752/t0(0) o8->oak-OST0002-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517260391 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jan 29 13:13:11 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 185 previous similar messages Jan 29 13:14:28 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 47 seconds Jan 29 13:14:28 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 171 previous similar messages Jan 29 13:23:18 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517260970/real 1517260998] req@ffff88027dc95200 x1590957017709184/t0(0) o8->oak-OST0014-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517261026 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 13:23:18 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 188 previous similar messages Jan 29 13:24:34 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 3 seconds Jan 29 13:24:34 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 183 previous similar messages Jan 29 13:33:24 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517261595/real 1517261604] req@ffff8800baa06100 x1590957017739920/t0(0) o8->oak-OST0000-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517261626 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 13:33:24 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 210 previous similar messages Jan 29 13:34:40 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 10 seconds Jan 29 13:34:40 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 204 previous similar messages Jan 29 13:43:56 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1517262195/real 1517262236] req@ffff88022a3c2700 x1590957017782720/t0(0) o8->oak-OST000e-osc-ffff88040df08000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1517262251 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jan 29 13:43:56 oak-gw04 kernel: Lustre: 9085:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 188 previous similar messages Jan 29 13:44:46 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 31 seconds Jan 29 13:44:46 oak-gw04 kernel: LNet: 1562:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 187 previous similar messages Jan 29 13:48:05 oak-gw04 kernel: Lustre: oak-OST002a-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 13:48:05 oak-gw04 kernel: Lustre: Skipped 6518 previous similar messages Jan 29 13:48:05 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 13:48:05 oak-gw04 kernel: Lustre: Skipped 18 previous similar messages Jan 29 13:48:21 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 13:48:21 oak-gw04 kernel: Lustre: Skipped 2066 previous similar messages Jan 29 13:48:29 oak-gw04 kernel: LustreError: 1563:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8803cb643a00 Jan 29 13:48:29 oak-gw04 kernel: LustreError: 1564:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8803cb643a00 Jan 29 13:48:29 oak-gw04 kernel: LustreError: 1563:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8803cb643a00 Jan 29 13:48:29 oak-gw04 kernel: LustreError: 1563:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8803cb643a00 Jan 29 13:48:29 oak-gw04 kernel: LustreError: 1564:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8803cb643a00 Jan 29 13:48:37 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 13:48:37 oak-gw04 kernel: Lustre: Skipped 3908 previous similar messages Jan 29 13:48:53 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 13:48:53 oak-gw04 kernel: Lustre: Skipped 3233 previous similar messages Jan 29 13:49:41 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 13:49:41 oak-gw04 kernel: Lustre: Skipped 5383 previous similar messages Jan 29 13:49:57 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 13:49:57 oak-gw04 kernel: Lustre: Skipped 5536 previous similar messages Jan 29 13:51:37 oak-gw04 kernel: Lustre: DEBUG MARKER: Mon Jan 29 13:51:37 2018 Jan 29 13:52:06 oak-gw04 kernel: Lustre: oak-OST0002-osc-ffff88040df08000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jan 29 13:52:06 oak-gw04 kernel: Lustre: Skipped 9497 previous similar messages Jan 29 13:52:06 oak-gw04 kernel: LustreError: 9319:0:(osc_quota.c:291:osc_quotactl()) ptlrpc_queue_wait failed, rc: -107 Jan 29 13:52:06 oak-gw04 kernel: LustreError: 9319:0:(osc_quota.c:291:osc_quotactl()) Skipped 10 previous similar messages Jan 29 13:52:06 oak-gw04 kernel: LustreError: 9598:0:(osc_quota.c:291:osc_quotactl()) ptlrpc_queue_wait failed, rc: -107 Jan 29 13:52:06 oak-gw04 kernel: LustreError: 9598:0:(osc_quota.c:291:osc_quotactl()) Skipped 11 previous similar messages Jan 29 13:52:31 oak-gw04 kernel: LustreError: 9394:0:(osc_quota.c:291:osc_quotactl()) ptlrpc_queue_wait failed, rc: -107 Jan 29 13:52:31 oak-gw04 kernel: LustreError: 9394:0:(osc_quota.c:291:osc_quotactl()) Skipped 11 previous similar messages Jan 29 13:52:32 oak-gw04 kernel: LustreError: 9089:0:(osc_request.c:1033:check_write_rcs()) Unexpected # bytes transferred: 0 (requested 4194304) Jan 29 13:52:35 oak-gw04 kernel: LustreError: 9598:0:(osc_quota.c:291:osc_quotactl()) ptlrpc_queue_wait failed, rc: -107 Jan 29 13:52:35 oak-gw04 kernel: LustreError: 9598:0:(osc_quota.c:291:osc_quotactl()) Skipped 11 previous similar messages Jan 29 13:52:35 oak-gw04 kernel: Lustre: oak-OST001c-osc-ffff88040df08000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jan 29 13:52:35 oak-gw04 kernel: Lustre: Skipped 11029 previous similar messages Jan 29 13:52:35 oak-gw04 kernel: LNetError: 9677:0:(o2iblnd_cb.c:1289:kiblnd_connect_peer()) Can't resolve addr for 10.0.2.101@o2ib5: -13