[ 3075.204629] Lustre: soaked-OST0005-osc-MDT0000: Connection to soaked-OST0005 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 3076.220103] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638199/real 0] req@ffff99e81a179b00 x1731865206429632/t0(0) o13->soaked-OST000d-osc-MDT0000@192.168.1.105@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651638207 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3076.251765] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3076.262485] Lustre: soaked-OST000d-osc-MDT0000: Connection to soaked-OST000d (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 3076.281149] Lustre: Skipped 1 previous similar message [ 3078.660327] Lustre: 3519:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638198/real 0] req@ffff99ec3b172880 x1731865206417728/t0(0) o6->soaked-OST0001-osc-MDT0000@192.168.1.105@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651638210 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3082.191702] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [ 3082.203948] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.105@o2ib (18): c: 0, oc: 0, rc: 8 [ 3086.192100] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3090.192531] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 8 seconds [ 3090.203904] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 6 previous similar messages [ 3094.192917] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 12 seconds [ 3094.204380] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 3114.194870] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3114.206243] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 2 previous similar messages [ 3122.195642] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 2 seconds [ 3134.196817] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3134.208196] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [ 3150.198255] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 4 seconds [ 3150.209619] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [ 3186.201480] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 2 seconds [ 3186.212853] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 3250.206786] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 11 seconds [ 3250.218258] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 20 previous similar messages [ 3289.090975] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0009_UUID (at 192.168.1.105@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec476e1000, cur 1651638420 expire 1651638270 last 1651638193 [ 3378.216333] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 11 seconds [ 3378.227805] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 40 previous similar messages [ 3647.310943] Lustre: soaked-OST0009-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3682.495305] Lustre: soaked-OST0001-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3728.127038] Lustre: soaked-OST0005-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3774.565549] Lustre: soaked-OST000d-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3828.061021] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3828.075426] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3828.575359] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3828.589750] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 66 previous similar messages [ 3828.601052] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3828.615918] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 66 previous similar messages [ 3829.599072] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3829.613459] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 159 previous similar messages [ 3829.624849] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3829.639711] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 159 previous similar messages [ 3831.602684] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3831.617073] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 317 previous similar messages [ 3831.628449] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3831.643311] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 317 previous similar messages [ 3835.632958] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3835.647360] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 525 previous similar messages [ 3835.658755] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3835.673618] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 525 previous similar messages [ 4204.234178] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4204.248571] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 643 previous similar messages [ 4204.259948] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4204.274812] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 643 previous similar messages [ 4580.417042] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4580.431435] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [ 4580.442932] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4580.457799] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [ 4960.220259] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4960.234657] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1715 previous similar messages [ 4960.246167] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4960.261033] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1715 previous similar messages [ 5335.585773] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 5335.600164] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1719 previous similar messages [ 5335.611649] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 5335.626514] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1719 previous similar messages [ 5711.893902] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 5711.908294] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1718 previous similar messages [ 5711.919767] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 5711.934631] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1718 previous similar messages [ 5814.198337] LustreError: 12981:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5814.698710] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5814.721440] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) Skipped 103 previous similar messages [ 5815.704685] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5815.727436] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) Skipped 215 previous similar messages [ 5819.937782] LustreError: 13023:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5819.960625] LustreError: 13023:0:(out_handler.c:910:out_tx_end()) Skipped 399 previous similar messages [ 5824.625183] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5824.647910] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 411 previous similar messages [ 5832.681030] LustreError: 12988:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5832.703848] LustreError: 12988:0:(out_handler.c:910:out_tx_end()) Skipped 1067 previous similar messages [ 5848.682997] LustreError: 12980:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5848.705824] LustreError: 12980:0:(out_handler.c:910:out_tx_end()) Skipped 3735 previous similar messages [ 6084.656079] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 6084.670472] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1718 previous similar messages [ 6084.681959] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 6084.696818] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1718 previous similar messages [ 6830.447411] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 6830.461838] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 3439 previous similar messages [ 6830.473345] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 6830.488216] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 3439 previous similar messages [ 6954.330738] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651642074/real 0] req@ffff99e80d9bba80 x1731865430302976/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651642081 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 6954.362399] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6954.373341] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6955.016776] Lustre: 3503:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651642075/real 0] req@ffff99e810b85580 x1731865430317440/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651642082 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 6955.048442] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6959.378955] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 19 seconds [ 6959.391201] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (20): c: 0, oc: 0, rc: 8 [ 6959.404983] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642071/real 1651642090] req@ffff99ec43130000 x1731865429951872/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642397 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6959.405034] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6959.456680] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6963.379120] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 7 seconds [ 6963.390485] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 54 previous similar messages [ 6963.400967] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642075/real 1651642094] req@ffff99ebfd0d3f00 x1731865430320000/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642401 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6963.433976] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6967.379454] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6971.379547] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642075/real 1651642102] req@ffff99e811077980 x1731865430359424/t0(0) o6->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642401 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6971.412554] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6995.380561] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [ 6995.391937] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 27 previous similar messages [ 7071.384047] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [ 7071.395426] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 17 previous similar messages [ 7141.693801] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0008_UUID (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e817084000, cur 1651642273 expire 1651642123 last 1651642046 [ 7141.718076] Lustre: Skipped 4 previous similar messages [ 7151.019626] Lustre: MGS: haven't heard from client 564d6672-71d3-4129-859c-bf1077267f7c (at 192.168.1.104@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e5b55f5c00, cur 1651642282 expire 1651642132 last 1651642053 [ 7151.043321] Lustre: Skipped 3 previous similar messages [ 7199.389962] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 5 seconds [ 7199.401336] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 32 previous similar messages [ 7575.844709] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 7575.859097] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 3436 previous similar messages [ 7575.870570] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 7575.885438] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 3436 previous similar messages [ 7760.568167] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7798.085168] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7843.083518] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7889.797579] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 8800.125040] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 8800.139478] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [ 8800.150969] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8800.165837] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [ 8940.885815] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 8940.900233] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 221 previous similar messages [ 8940.911616] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8940.926473] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 221 previous similar messages [ 9175.577672] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [ 9175.591884] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 215 previous similar messages [ 9175.603262] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [ 9175.617938] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 215 previous similar messages [ 9178.625121] LustreError: 3779:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e86cdf8200 [ 9178.636120] LustreError: 3779:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0002-osp-MDT0000: write updates failed: rc = -116 [ 9549.431970] Lustre: 3497:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651644669/real 0] req@ffff99e81333ad00 x1731865561876736/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651644676 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 9549.463638] Lustre: 3497:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9549.474356] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 9554.621168] Lustre: 3509:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651644668/real 0] req@ffff99ec2a970000 x1731865561876672/t0(0) o13->soaked-OST0006-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651644675 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 9554.652831] Lustre: 3509:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9554.663549] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 9554.682214] Lustre: Skipped 1 previous similar message [ 9555.501201] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [ 9555.513452] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (21): c: 0, oc: 0, rc: 8 [ 9559.501433] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 5 seconds [ 9559.512802] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 72 previous similar messages [ 9595.503193] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9595.514561] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 9671.506897] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9671.518264] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [ 9760.286734] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e8176dcc00, cur 1651644891 expire 1651644741 last 1651644664 [ 9799.514204] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9799.525572] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 33 previous similar messages [10071.526582] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [10071.537956] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 66 previous similar messages [10373.395804] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10400.410141] Lustre: mdt01_018: service thread pid 8315 was inactive for 200.154 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [10400.410145] Pid: 3776, comm: mdt00_006 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [10400.410150] Call Trace: [10400.410188] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [10400.410199] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [10400.410229] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [10400.410249] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [10400.410267] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [10400.410286] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [10400.410304] [<0>] lod_prepare_create+0x230/0x320 [lod] [10400.410322] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [10400.410339] [<0>] lod_declare_create+0x1f5/0x600 [lod] [10400.410360] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [10400.410374] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [10400.410388] [<0>] mdd_declare_create+0x66/0x480 [mdd] [10400.410401] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [10400.410437] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [10400.410463] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [10400.410484] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [10400.410507] [<0>] mdt_intent_open+0x93/0x480 [mdt] [10400.410529] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [10400.410550] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [10400.410620] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [10400.410683] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [10400.410766] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [10400.410844] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [10400.410914] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [10400.410982] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [10400.410988] [<0>] kthread+0xd1/0xe0 [10400.410994] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [10400.411025] [<0>] 0xfffffffffffffffe [10400.607430] Lustre: Skipped 1 previous similar message [10400.613182] Pid: 8315, comm: mdt01_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [10400.624192] Call Trace: [10400.626960] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [10400.633012] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [10400.638785] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [10400.644849] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [10400.651183] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [10400.658119] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [10400.664189] [<0>] lod_prepare_create+0x230/0x320 [lod] [10400.669969] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [10400.676413] [<0>] lod_declare_create+0x1f5/0x600 [lod] [10400.682183] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [10400.689400] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [10400.696522] [<0>] mdd_declare_create+0x66/0x480 [mdd] [10400.702190] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [10400.707302] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [10400.712878] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [10400.718069] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [10400.723845] [<0>] mdt_intent_open+0x93/0x480 [mdt] [10400.729237] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [10400.734615] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [10400.740337] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [10400.746318] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [10400.752699] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [10400.758006] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [10400.764188] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [10400.771235] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [10400.776714] [<0>] kthread+0xd1/0xe0 [10400.780628] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [10400.786404] [<0>] 0xfffffffffffffffe [10408.455288] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10453.530384] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10498.598006] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10590.982372] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [10590.996772] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 439 previous similar messages [10591.008179] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [10591.023036] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 439 previous similar messages [10862.708702] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [10862.723105] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 220 previous similar messages [10862.734496] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [10862.749360] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 220 previous similar messages [10862.816467] LustreError: 7035:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e86cdf8200 [10862.827461] LustreError: 7035:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0002-osp-MDT0000: write updates failed: rc = -116 [11315.467860] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11315.490583] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) Skipped 1583 previous similar messages [11315.526385] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [11315.540570] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 218 previous similar messages [11315.551941] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [11315.566602] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 218 previous similar messages [11324.458665] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11324.481403] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 1427 previous similar messages [11332.461796] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11332.484549] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 855 previous similar messages [11352.750142] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11352.772888] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 1423 previous similar messages [11788.396746] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [11788.410958] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 742 previous similar messages [11788.422331] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [11788.436998] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 742 previous similar messages [11875.168598] LustreError: 7673:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11875.191340] LustreError: 7673:0:(out_handler.c:910:out_tx_end()) Skipped 2039 previous similar messages [11941.802780] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11941.825521] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 7487 previous similar messages [11979.038147] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647103/real 0] req@ffff99ec40b08900 x1731865633767488/t0(0) o13->soaked-OST0007-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647110 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11979.069836] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [11979.080671] Lustre: soaked-OST0007-osc-MDT0000: Connection to soaked-OST0007 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11979.099336] Lustre: Skipped 1 previous similar message [11979.958137] Lustre: 3498:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647104/real 0] req@ffff99e80eef1680 x1731865633769024/t0(0) o13->soaked-OST000b-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647111 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11979.989806] Lustre: soaked-OST000b-osc-MDT0000: Connection to soaked-OST000b (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11981.686236] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647105/real 0] req@ffff99ec40b0ad00 x1731865633770624/t0(0) o13->soaked-OST000f-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647112 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11981.717907] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11981.728622] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11981.747312] Lustre: Skipped 1 previous similar message [11988.534575] Lustre: 3494:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647103/real 0] req@ffff99e7f4e60000 x1731865633767616/t0(0) o400->soaked-OST0007-osc-MDT0000@192.168.1.107@o2ib:28/4 lens 224/224 e 0 to 1 dl 1651647119 ref 2 fl Rpc:XNr/0/ffffffff rc 0/-1 job:'' [11991.609724] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [11991.621979] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.107@o2ib (20): c: 0, oc: 0, rc: 8 [11995.609846] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 7 seconds [11995.621218] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 6 previous similar messages [12063.612911] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 2 seconds [12063.624290] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [12184.485380] Lustre: MGS: haven't heard from client 61a79e1f-9f76-41dc-8d35-5d871b03db11 (at 192.168.1.107@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e86b913000, cur 1651647315 expire 1651647165 last 1651647088 [12184.509104] Lustre: Skipped 4 previous similar messages [12191.618627] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 13 seconds [12191.630108] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 39 previous similar messages [12198.226254] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0003_UUID (at 192.168.1.107@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec6537b400, cur 1651647329 expire 1651647179 last 1651647102 [12723.832985] Lustre: soaked-OST000b-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12748.382012] Lustre: soaked-OST0003-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12799.756837] Lustre: soaked-OST000f-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12843.191517] Lustre: soaked-OST0007-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12920.045170] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [12920.059562] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 452 previous similar messages [12920.070938] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [12920.085801] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 452 previous similar messages [13045.808055] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13045.822455] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 450 previous similar messages [13045.833840] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13045.848699] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 450 previous similar messages [13169.831601] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13169.845993] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 553 previous similar messages [13169.857383] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13169.872242] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 553 previous similar messages [13401.515261] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13401.529678] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 551 previous similar messages [13401.541069] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13401.555932] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 551 previous similar messages [13978.949869] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13978.964259] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1114 previous similar messages [13978.975748] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13978.990606] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1114 previous similar messages [13979.998125] LustreError: 8315:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [13980.009142] LustreError: 8315:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [14184.243025] LustreError: 3844:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [14184.254038] LustreError: 3844:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [14738.875746] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14738.898478] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 2515 previous similar messages [14754.967381] LustreError: 7630:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14754.990105] LustreError: 7630:0:(out_handler.c:910:out_tx_end()) Skipped 811 previous similar messages [14787.134232] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14787.156964] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 1919 previous similar messages [15259.056801] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [15259.071213] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1074 previous similar messages [15259.082708] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [15259.097583] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1074 previous similar messages [15832.070105] perf: interrupt took too long (2604 > 2500), lowering kernel.perf_event_max_sample_rate to 76000 [16358.703277] Lustre: 3522:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651651482/real 0] req@ffff99ebfcf5f980 x1731865750022080/t0(0) o13->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651651489 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [16358.734937] Lustre: 3522:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16358.745666] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [16360.490380] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651651484/real 0] req@ffff99e80dce9f80 x1731865750051008/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651651491 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [16360.522050] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [16367.795698] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [16367.807959] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (18): c: 0, oc: 0, rc: 8 [16367.821714] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651651479/real 1651651498] req@ffff99ebdf6ccc80 x1731865749975424/t0(0) o6->soaked-OST0006-osc-MDT0000@192.168.1.102@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651651523 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [16367.854730] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [16556.106656] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST000a_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec6ce69800, cur 1651651687 expire 1651651537 last 1651651460 [16556.130948] Lustre: Skipped 3 previous similar messages [16571.119985] Lustre: MGS: haven't heard from client bfe70c3a-93cf-4893-ab6c-fd51acbe2210 (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80f20b000, cur 1651651702 expire 1651651552 last 1651651475 [16571.143701] Lustre: Skipped 3 previous similar messages [16683.808689] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [16683.820074] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 68 previous similar messages [16715.810133] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 4 seconds [16715.821499] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 15 previous similar messages [16779.812995] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 6 seconds [16779.824361] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 29 previous similar messages [16965.538292] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17009.993306] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17054.930575] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17099.890548] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17113.228709] LustreError: 11-0: soaked-OST0002-osc-MDT0000: operation ost_destroy to node 192.168.1.107@o2ib failed: rc = -107 [17113.228737] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17113.228740] Lustre: Skipped 2 previous similar messages [17113.265880] LustreError: Skipped 2 previous similar messages [17120.185656] LustreError: 11-0: soaked-OST000a-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [17120.198251] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17124.827923] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17125.624354] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [17129.962124] LustreError: 11-0: soaked-OST000e-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [17129.974718] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17134.376998] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [17134.388499] Lustre: Skipped 1 previous similar message [17168.942311] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17168.956690] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 974 previous similar messages [17168.968062] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17168.982925] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 974 previous similar messages [17178.353542] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17178.367928] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1235 previous similar messages [17178.379397] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17178.394255] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1235 previous similar messages [17403.601697] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [17403.624412] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) Skipped 1487 previous similar messages [17412.485548] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [17412.508279] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 899 previous similar messages [17553.230285] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17553.244687] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 43 previous similar messages [17553.255988] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17553.270848] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 43 previous similar messages [17556.971065] LustreError: 3775:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17556.982074] LustreError: 3775:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17557.723321] LustreError: 3776:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17557.734329] LustreError: 3776:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17559.111327] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17559.122328] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) Skipped 1 previous similar message [17559.132758] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17559.145871] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) Skipped 1 previous similar message [17561.275415] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17561.286409] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17781.962360] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17781.976765] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1514 previous similar messages [17781.988260] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17782.003130] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1514 previous similar messages [18001.950533] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [18001.964937] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1237 previous similar messages [18001.976438] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [18001.991307] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1237 previous similar messages [18356.494387] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18356.517127] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 2799 previous similar messages [18358.519508] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18358.542239] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) Skipped 379 previous similar messages [18362.700751] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18362.723515] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 599 previous similar messages [18370.755776] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18370.778517] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 419 previous similar messages [18386.818212] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18386.840955] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 1259 previous similar messages [18818.493369] Lustre: 3492:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651653942/real 0] req@ffff99e7ca8d0000 x1731865846555840/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651653949 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [18818.525050] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [18819.933437] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651653943/real 0] req@ffff99e7ca8d3600 x1731865846624768/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651653950 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [18819.965113] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18819.975853] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [18819.994522] Lustre: Skipped 1 previous similar message [18827.904803] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [18827.917064] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (19): c: 0, oc: 0, rc: 8 [19018.611368] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0008_UUID (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e816611800, cur 1651654149 expire 1651653999 last 1651653922 [19020.073179] Lustre: MGS: haven't heard from client ea8ea456-a9d5-467c-8526-0377b4b867c8 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99ec46611c00, cur 1651654150 expire 1651654000 last 1651653922 [19020.096878] Lustre: Skipped 3 previous similar messages [19022.862584] Lustre: mdt00_015: service thread pid 8312 was inactive for 200.668 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19022.862587] Lustre: mdt00_003: service thread pid 3654 was inactive for 200.694 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19022.862592] Lustre: mdt01_018: service thread pid 8315 was inactive for 200.678 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19022.862597] Pid: 3787, comm: mdt01_011 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19022.862601] Lustre: Skipped 1 previous similar message [19022.862604] Lustre: Skipped 1 previous similar message [19022.862610] Call Trace: [19022.862653] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19022.862664] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19022.862698] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19022.862716] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19022.862734] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19022.862751] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19022.862768] [<0>] lod_prepare_create+0x230/0x320 [lod] [19022.862783] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19022.862799] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19022.862823] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19022.862835] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19022.862848] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19022.862861] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19022.862901] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19022.862925] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19022.862945] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19022.862965] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19022.862985] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19022.863006] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19022.863080] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19022.863141] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19022.863223] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19022.863296] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19022.863361] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19022.863424] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19022.863431] [<0>] kthread+0xd1/0xe0 [19022.863438] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19022.863469] [<0>] 0xfffffffffffffffe [19022.863471] Pid: 8315, comm: mdt01_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19022.863472] Call Trace: [19022.863502] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19022.863512] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19022.863559] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19022.863579] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19022.863598] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19022.863616] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19022.863634] [<0>] lod_prepare_create+0x230/0x320 [lod] [19022.863652] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19022.863669] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19022.863687] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19022.863701] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19022.863714] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19022.863728] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19022.863757] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19022.863783] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19022.863805] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19022.863827] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19022.863849] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19022.863871] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19022.863925] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19022.863988] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19022.864066] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19022.864143] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19022.864213] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19022.864281] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19022.864285] [<0>] kthread+0xd1/0xe0 [19022.864289] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19022.864300] [<0>] 0xfffffffffffffffe [19023.284154] Pid: 8312, comm: mdt00_015 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19023.295135] Call Trace: [19023.297901] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19023.303946] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19023.309703] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19023.315746] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19023.322077] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19023.328993] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19023.335029] [<0>] lod_prepare_create+0x230/0x320 [lod] [19023.340779] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19023.347210] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19023.352961] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19023.360171] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19023.367268] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19023.372919] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19023.378010] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19023.383565] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19023.388732] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19023.394485] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19023.399843] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19023.405206] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19023.410901] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19023.416864] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19023.423224] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19023.428517] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19023.434682] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19023.441712] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19023.447172] [<0>] kthread+0xd1/0xe0 [19023.451070] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19023.456823] [<0>] 0xfffffffffffffffe [19081.841301] INFO: task mdt01_002:3565 blocked for more than 120 seconds. [19081.848805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19081.857558] mdt01_002 D ffff99ec4e59b180 0 3565 2 0x00000080 [19081.865503] Call Trace: [19081.868270] [] schedule+0x29/0x70 [19081.873843] [] rwsem_down_write_failed+0x215/0x3c0 [19081.881056] [] call_rwsem_down_write_failed+0x17/0x30 [19081.888556] [] down_write+0x2d/0x3d [19081.894347] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19081.902921] [] ? osp_statfs+0x1ff/0x530 [osp] [19081.909674] [] ? mutex_lock+0x12/0x2f [19081.915634] [] ? memset+0x22/0xb0 [19081.921209] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19081.928855] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19081.937430] [] lod_prepare_create+0x230/0x320 [lod] [19081.944740] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19081.952730] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19081.960541] [] lod_declare_create+0x1f5/0x600 [lod] [19081.967866] [] ? lod_get_ea+0xc6/0x530 [lod] [19081.974507] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19081.983266] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19081.991945] [] mdd_declare_create+0x66/0x480 [mdd] [19081.999153] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.005807] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.012973] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.020800] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.029176] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.037751] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.044478] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.051805] [] mdt_intent_open+0x93/0x480 [mdt] [19082.058728] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.065660] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.073648] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.080924] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.088424] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.096410] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.103553] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.111496] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.119258] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.126116] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.133847] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.142317] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.150231] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.158794] [] ? task_rq_unlock+0x20/0x20 [19082.165124] [] ? __wake_up+0x13/0x20 [19082.171006] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.178058] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.185747] [] kthread+0xd1/0xe0 [19082.191204] [] ? insert_kthread_work+0x40/0x40 [19082.198030] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.205338] [] ? insert_kthread_work+0x40/0x40 [19082.212165] INFO: task mdt01_004:3769 blocked for more than 120 seconds. [19082.219657] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.228410] mdt01_004 D ffff99ec4769e300 0 3769 2 0x00000080 [19082.236326] Call Trace: [19082.239073] [] schedule+0x29/0x70 [19082.244626] [] rwsem_down_write_failed+0x215/0x3c0 [19082.251829] [] call_rwsem_down_write_failed+0x17/0x30 [19082.259326] [] down_write+0x2d/0x3d [19082.265085] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.273694] [] ? tgt_free_reply_data+0xe7/0x270 [ptlrpc] [19082.281481] [] ? kfree+0x106/0x140 [19082.287142] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19082.294749] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19082.303327] [] lod_prepare_create+0x230/0x320 [lod] [19082.310635] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19082.318621] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19082.326415] [] lod_declare_create+0x1f5/0x600 [lod] [19082.333726] [] ? lod_get_ea+0xc6/0x530 [lod] [19082.340354] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19082.349109] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19082.357775] [] mdd_declare_create+0x66/0x480 [mdd] [19082.364980] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.371622] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.378753] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.386559] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.394936] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.403509] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.410253] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.417556] [] mdt_intent_open+0x93/0x480 [mdt] [19082.424479] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.431402] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.439382] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.446615] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.454115] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.462096] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.469238] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.477163] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.484897] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.491755] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.499489] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.507956] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.515883] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.524443] [] ? task_rq_unlock+0x20/0x20 [19082.530772] [] ? __wake_up+0x13/0x20 [19082.536654] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.543699] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.551378] [] kthread+0xd1/0xe0 [19082.556835] [] ? insert_kthread_work+0x40/0x40 [19082.563650] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.570949] [] ? insert_kthread_work+0x40/0x40 [19082.577771] INFO: task mdt01_006:3774 blocked for more than 120 seconds. [19082.585264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.594015] mdt01_006 D ffff99ec6a4b6300 0 3774 2 0x00000080 [19082.601938] Call Trace: [19082.604685] [] schedule+0x29/0x70 [19082.610240] [] rwsem_down_write_failed+0x215/0x3c0 [19082.617442] [] call_rwsem_down_write_failed+0x17/0x30 [19082.624936] [] down_write+0x2d/0x3d [19082.630696] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.639267] [] ? lod_sub_write+0x1d0/0x440 [lod] [19082.646305] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19082.653905] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19082.662474] [] lod_prepare_create+0x230/0x320 [lod] [19082.669787] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19082.677773] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19082.685568] [] lod_declare_create+0x1f5/0x600 [lod] [19082.692872] [] ? lod_get_ea+0xc6/0x530 [lod] [19082.699499] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19082.708275] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19082.716939] [] mdd_declare_create+0x66/0x480 [mdd] [19082.724143] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.730785] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.737912] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.745719] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.754094] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.762664] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.769391] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.776700] [] mdt_intent_open+0x93/0x480 [mdt] [19082.783622] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.790544] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.798533] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.805777] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.813309] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.821323] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.828497] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.836433] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.844181] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.851054] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.858798] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.867281] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.875227] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.883813] [] ? task_rq_unlock+0x20/0x20 [19082.890163] [] ? __wake_up+0x13/0x20 [19082.896061] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.903121] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.910825] [] kthread+0xd1/0xe0 [19082.916298] [] ? insert_kthread_work+0x40/0x40 [19082.923129] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.930450] [] ? insert_kthread_work+0x40/0x40 [19082.937272] INFO: task mdt00_005:3775 blocked for more than 120 seconds. [19082.944772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.953523] mdt00_005 D ffff99ec6a4b2100 0 3775 2 0x00000080 [19082.961460] Call Trace: [19082.964214] [] schedule+0x29/0x70 [19082.969775] [] rwsem_down_write_failed+0x215/0x3c0 [19082.976987] [] call_rwsem_down_write_failed+0x17/0x30 [19082.984490] [] down_write+0x2d/0x3d [19082.990289] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.998860] [] ? lod_sub_write+0x1d0/0x440 [lod] [19083.005900] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.013526] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.022119] [] lod_prepare_create+0x230/0x320 [lod] [19083.029435] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.037438] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.045260] [] lod_declare_create+0x1f5/0x600 [lod] [19083.052581] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.059232] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.068017] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.076687] [] mdd_declare_create+0x66/0x480 [mdd] [19083.083909] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.090559] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.097698] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.105513] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.113905] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.122479] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.129208] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.136526] [] mdt_intent_open+0x93/0x480 [mdt] [19083.143472] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.150410] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.158414] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.165655] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.173169] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.181166] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.188313] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.196269] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.204026] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.210907] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.218651] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.227134] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.235060] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.243637] [] ? task_rq_unlock+0x20/0x20 [19083.249975] [] ? __wake_up+0x13/0x20 [19083.255879] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.262949] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.270659] [] kthread+0xd1/0xe0 [19083.276137] [] ? insert_kthread_work+0x40/0x40 [19083.282965] [] ret_from_fork_nospec_begin+0x21/0x21 [19083.290283] [] ? insert_kthread_work+0x40/0x40 [19083.297113] INFO: task mdt00_007:3780 blocked for more than 120 seconds. [19083.304612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19083.313370] mdt00_007 D ffff99ec5d1ab180 0 3780 2 0x00000080 [19083.321335] Call Trace: [19083.324090] [] schedule+0x29/0x70 [19083.329652] [] rwsem_down_write_failed+0x215/0x3c0 [19083.336862] [] call_rwsem_down_write_failed+0x17/0x30 [19083.344365] [] down_write+0x2d/0x3d [19083.350138] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19083.358706] [] ? __radix_tree_lookup+0x84/0xf0 [19083.365545] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.373154] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.381738] [] lod_prepare_create+0x230/0x320 [lod] [19083.389066] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.397075] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.404875] [] lod_declare_create+0x1f5/0x600 [lod] [19083.412195] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.418841] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.427608] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.436302] [] mdd_declare_create+0x66/0x480 [mdd] [19083.443533] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.450193] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.457324] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.465131] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.473519] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.482100] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.488828] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.496154] [] mdt_intent_open+0x93/0x480 [mdt] [19083.503099] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.510030] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.518041] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.525308] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.532839] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.540836] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.547992] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.555928] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.563675] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.570557] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.578314] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.586789] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.594715] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.603290] [] ? task_rq_unlock+0x20/0x20 [19083.609617] [] ? __wake_up+0x13/0x20 [19083.615498] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.622544] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.630238] [] kthread+0xd1/0xe0 [19083.635701] [] ? insert_kthread_work+0x40/0x40 [19083.642516] [] ret_from_fork_nospec_begin+0x21/0x21 [19083.649822] [] ? insert_kthread_work+0x40/0x40 [19083.656635] INFO: task mdt00_009:4645 blocked for more than 120 seconds. [19083.664133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19083.672890] mdt00_009 D ffff99ec6941d280 0 4645 2 0x00000080 [19083.680817] Call Trace: [19083.683552] [] schedule+0x29/0x70 [19083.689113] [] rwsem_down_write_failed+0x215/0x3c0 [19083.696330] [] call_rwsem_down_write_failed+0x17/0x30 [19083.703821] [] down_write+0x2d/0x3d [19083.709566] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19083.718110] [] ? __radix_tree_lookup+0x84/0xf0 [19083.724939] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.732538] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.741114] [] lod_prepare_create+0x230/0x320 [lod] [19083.748428] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.756428] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.764215] [] lod_declare_create+0x1f5/0x600 [lod] [19083.771514] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.778140] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.786922] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.795579] [] mdd_declare_create+0x66/0x480 [mdd] [19083.802780] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.809421] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.816547] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.824366] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.832753] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.841328] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.848079] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.855404] [] mdt_intent_open+0x93/0x480 [mdt] [19083.862322] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.869239] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.877214] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.884441] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.891936] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.899917] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.907054] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.914969] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.922697] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.929551] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.937280] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.945745] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.953660] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.962218] [] ? task_rq_unlock+0x20/0x20 [19083.968541] [] ? __wake_up+0x13/0x20 [19083.974418] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.981460] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.989149] [] kthread+0xd1/0xe0 [19083.994600] [] ? insert_kthread_work+0x40/0x40 [19084.001423] [] ret_from_fork_nospec_begin+0x21/0x21 [19084.008719] [] ? insert_kthread_work+0x40/0x40 [19107.917457] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [19107.928833] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 60 previous similar messages [19131.918500] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [19131.929865] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [19157.524661] Lustre: mdt00_009: service thread pid 4645 was inactive for 242.485 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19159.060697] Lustre: mdt01_002: service thread pid 3565 was inactive for 244.022 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19163.919958] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 30 seconds [19163.931421] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 19 previous similar messages [19164.180963] Lustre: mdt00_007: service thread pid 3780 was inactive for 242.018 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19204.020806] INFO: task mdt01_002:3565 blocked for more than 120 seconds. [19204.028319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.037080] mdt01_002 D ffff99ec4e59b180 0 3565 2 0x00000080 [19204.045020] Call Trace: [19204.047786] [] schedule+0x29/0x70 [19204.053364] [] rwsem_down_write_failed+0x215/0x3c0 [19204.060594] [] call_rwsem_down_write_failed+0x17/0x30 [19204.068100] [] down_write+0x2d/0x3d [19204.073887] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.082461] [] ? osp_statfs+0x1ff/0x530 [osp] [19204.089206] [] ? mutex_lock+0x12/0x2f [19204.095171] [] ? memset+0x22/0xb0 [19204.100745] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.108386] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.116956] [] lod_prepare_create+0x230/0x320 [lod] [19204.124270] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.132286] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.140083] [] lod_declare_create+0x1f5/0x600 [lod] [19204.147399] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.154052] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.162795] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.171467] [] mdd_declare_create+0x66/0x480 [mdd] [19204.178672] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.185366] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.192545] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.200373] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.208761] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.217333] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.224060] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.231362] [] mdt_intent_open+0x93/0x480 [mdt] [19204.238287] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.245209] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.253193] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.260470] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.267972] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19204.275959] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19204.283093] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19204.291020] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19204.298753] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19204.305616] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19204.313325] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19204.321790] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19204.329722] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19204.338307] [] ? task_rq_unlock+0x20/0x20 [19204.344635] [] ? __wake_up+0x13/0x20 [19204.350512] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19204.357561] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19204.365264] [] kthread+0xd1/0xe0 [19204.370718] [] ? insert_kthread_work+0x40/0x40 [19204.377555] [] ret_from_fork_nospec_begin+0x21/0x21 [19204.384852] [] ? insert_kthread_work+0x40/0x40 [19204.391674] INFO: task mdt01_004:3769 blocked for more than 120 seconds. [19204.399171] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.407931] mdt01_004 D ffff99ec4769e300 0 3769 2 0x00000080 [19204.415867] Call Trace: [19204.418607] [] schedule+0x29/0x70 [19204.424160] [] rwsem_down_write_failed+0x215/0x3c0 [19204.431374] [] call_rwsem_down_write_failed+0x17/0x30 [19204.438866] [] down_write+0x2d/0x3d [19204.444635] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.453232] [] ? tgt_free_reply_data+0xe7/0x270 [ptlrpc] [19204.461022] [] ? kfree+0x106/0x140 [19204.466678] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.474283] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.482863] [] lod_prepare_create+0x230/0x320 [lod] [19204.490168] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.498164] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.505958] [] lod_declare_create+0x1f5/0x600 [lod] [19204.513275] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.519899] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.528656] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.537314] [] mdd_declare_create+0x66/0x480 [mdd] [19204.544519] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.551157] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.558276] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.566074] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.574447] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.583000] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.589726] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.597031] [] mdt_intent_open+0x93/0x480 [mdt] [19204.603950] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.610868] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.618854] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.626077] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.633573] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19204.641553] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19204.648681] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19204.656588] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19204.664299] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19204.671144] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19204.678863] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19204.687328] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19204.695240] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19204.703798] [] ? task_rq_unlock+0x20/0x20 [19204.710117] [] ? __wake_up+0x13/0x20 [19204.715986] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19204.723023] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19204.730706] [] kthread+0xd1/0xe0 [19204.736161] [] ? insert_kthread_work+0x40/0x40 [19204.742970] [] ret_from_fork_nospec_begin+0x21/0x21 [19204.750281] [] ? insert_kthread_work+0x40/0x40 [19204.757096] INFO: task mdt01_005:3770 blocked for more than 120 seconds. [19204.764600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.773358] mdt01_005 D ffff99ec6a533180 0 3770 2 0x00000080 [19204.781312] Call Trace: [19204.784036] [] schedule+0x29/0x70 [19204.789602] [] rwsem_down_write_failed+0x215/0x3c0 [19204.796810] [] call_rwsem_down_write_failed+0x17/0x30 [19204.804307] [] down_write+0x2d/0x3d [19204.810063] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.818622] [] ? osp_statfs+0x1ff/0x530 [osp] [19204.825359] [] ? mutex_lock+0x12/0x2f [19204.831324] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.838915] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.847493] [] lod_prepare_create+0x230/0x320 [lod] [19204.854799] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.862777] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.870578] [] lod_declare_create+0x1f5/0x600 [lod] [19204.877884] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.884509] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.893262] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.901920] [] mdd_declare_create+0x66/0x480 [mdd] [19204.909125] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.915760] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.922884] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.930692] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.939051] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.947615] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.954338] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.961663] [] mdt_intent_open+0x93/0x480 [mdt] [19204.968577] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.975508] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.983490] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.990729] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.998222] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19205.006214] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19205.013336] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19205.021250] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19205.028967] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19205.035806] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19205.043539] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19205.052010] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19205.059904] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19205.068461] [] ? task_rq_unlock+0x20/0x20 [19205.074791] [] ? __wake_up+0x13/0x20 [19205.080655] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19205.087710] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19205.095393] [] kthread+0xd1/0xe0 [19205.100853] [] ? insert_kthread_work+0x40/0x40 [19205.107668] [] ret_from_fork_nospec_begin+0x21/0x21 [19205.114979] [] ? insert_kthread_work+0x40/0x40 [19205.121791] INFO: task mdt01_006:3774 blocked for more than 120 seconds. [19205.129289] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19205.138040] mdt01_006 D ffff99ec6a4b6300 0 3774 2 0x00000080 [19205.145928] Call Trace: [19205.148680] [] schedule+0x29/0x70 [19205.154231] [] rwsem_down_write_failed+0x215/0x3c0 [19205.161430] [] call_rwsem_down_write_failed+0x17/0x30 [19205.168923] [] down_write+0x2d/0x3d [19205.174685] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19205.183255] [] ? lod_sub_write+0x1d0/0x440 [lod] [19205.190285] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19205.197882] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19205.206444] [] lod_prepare_create+0x230/0x320 [lod] [19205.213759] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19205.221732] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19205.229525] [] lod_declare_create+0x1f5/0x600 [lod] [19205.236827] [] ? lod_get_ea+0xc6/0x530 [lod] [19205.243463] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19205.252208] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19205.260873] [] mdd_declare_create+0x66/0x480 [mdd] [19205.268078] [] mdd_create+0x9a9/0x1cd0 [mdd] [19205.274726] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19205.281837] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19205.289644] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19205.298012] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19205.306589] [] mdt_reint_rec+0x8a/0x240 [mdt] [19205.313324] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19205.320628] [] mdt_intent_open+0x93/0x480 [mdt] [19205.327566] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19205.334487] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19205.342490] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19205.349708] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19205.357194] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19205.365166] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19205.372297] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19205.380190] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19205.387916] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19205.394781] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19205.402533] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19205.410991] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19205.418896] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19205.427475] [] ? task_rq_unlock+0x20/0x20 [19205.433809] [] ? __wake_up+0x13/0x20 [19205.439702] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19205.446732] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19205.454433] [] kthread+0xd1/0xe0 [19205.459900] [] ? insert_kthread_work+0x40/0x40 [19205.466740] [] ret_from_fork_nospec_begin+0x21/0x21 [19205.474031] [] ? insert_kthread_work+0x40/0x40 [19227.922847] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 94 seconds [19227.934308] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [19330.076436] ptlrpc_watchdog_fire: 10 callbacks suppressed [19330.082524] Lustre: mdt01_010: service thread pid 3785 was inactive for 344.744 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19330.103418] Pid: 3785, comm: mdt01_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19330.114410] Call Trace: [19330.117181] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19330.123160] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19330.130193] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19330.136253] [<0>] lod_prepare_create+0x230/0x320 [lod] [19330.142030] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19330.148487] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19330.154289] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19330.161496] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19330.168606] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19330.174254] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19330.179368] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19330.184928] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19330.190112] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19330.195877] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19330.201239] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19330.206607] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19330.212340] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19330.218313] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19330.224698] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19330.230013] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19330.236208] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19330.243237] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19330.248690] [<0>] kthread+0xd1/0xe0 [19330.252588] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19330.258347] [<0>] 0xfffffffffffffffe [19416.628368] Lustre: 22106:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec2ba3b180 x1731763470721664/t0(0) o101->00629187-5c6d-4754-a616-e35b213076e9@192.168.1.121@o2ib:697/0 lens 1312/4072 e 23 to 0 dl 1651654552 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19417.630406] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebe87a0900 x1731763170365440/t0(0) o101->9a274bc2-9a2c-4748-bcc9-e7338953ee88@192.168.1.125@o2ib:698/0 lens 1312/4072 e 23 to 0 dl 1651654553 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19417.662054] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [19423.260103] Lustre: soaked-MDT0000: Client 24816f25-f323-4fbc-a2f1-9bdf8a245040 (at 192.168.1.128@o2ib) reconnecting [19423.271875] Lustre: Skipped 2 previous similar messages [19467.298589] Lustre: mdt01_015: service thread pid 7035 was inactive for 444.876 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19467.298591] Lustre: mdt01_014: service thread pid 5289 was inactive for 444.875 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19467.298594] Pid: 22107, comm: mdt01_021 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19467.298596] Lustre: Skipped 5 previous similar messages [19467.298597] Call Trace: [19467.298643] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19467.298684] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19467.298700] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19467.298712] [<0>] lod_prepare_create+0x230/0x320 [lod] [19467.298723] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19467.298734] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19467.298757] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19467.298775] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19467.298796] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19467.298804] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19467.298836] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19467.298851] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19467.298863] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19467.298876] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19467.298889] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19467.298901] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19467.298971] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19467.299019] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19467.299098] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19467.299148] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19467.299193] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19467.299253] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19467.299258] [<0>] kthread+0xd1/0xe0 [19467.299263] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19467.299312] [<0>] 0xfffffffffffffffe [19467.499733] Lustre: Skipped 1 previous similar message [19467.505483] Pid: 7035, comm: mdt01_015 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19467.516460] Call Trace: [19467.519206] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19467.525154] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19467.532165] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19467.538203] [<0>] lod_prepare_create+0x230/0x320 [lod] [19467.543949] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19467.550376] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19467.556131] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19467.563330] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19467.570439] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19467.576088] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19467.581178] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19467.586736] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19467.591902] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19467.597656] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19467.603019] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19467.608383] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19467.614091] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19467.620064] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19467.626430] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19467.631726] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19467.637878] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19467.644908] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19467.650362] [<0>] kthread+0xd1/0xe0 [19467.654263] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19467.660024] [<0>] 0xfffffffffffffffe [19509.668553] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebf29b3f00 x1731763737452864/t0(0) o101->cb2a3a98-2672-400e-b628-aaad60526b53@192.168.1.137@o2ib:35/0 lens 1304/4072 e 5 to 0 dl 1651654645 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19516.107655] Lustre: soaked-MDT0000: Client cb2a3a98-2672-400e-b628-aaad60526b53 (at 192.168.1.137@o2ib) reconnecting [19516.119588] Lustre: Skipped 2 previous similar messages [19516.686837] Lustre: 3693:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebe73e8d80 x1731770169837504/t0(0) o101->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:42/0 lens 1312/4072 e 4 to 0 dl 1651654652 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19516.718285] Lustre: 3693:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [19523.232257] Lustre: soaked-MDT0000: Client 2b5be8a8-9023-43d6-8098-538f23791711 (at 192.168.1.122@o2ib) reconnecting [19523.244038] Lustre: Skipped 1 previous similar message [19617.961465] Lustre: 35179:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb78c4f980 x1731764682509312/t0(0) o101->26c1c6f4-8570-40b8-af2d-5b5a5e2c346b@192.168.1.123@o2ib:143/0 lens 1312/4072 e 2 to 0 dl 1651654753 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19617.993112] Lustre: 35179:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [19623.428848] Lustre: soaked-MDT0000: Client 26c1c6f4-8570-40b8-af2d-5b5a5e2c346b (at 192.168.1.123@o2ib) reconnecting [19623.440640] Lustre: Skipped 1 previous similar message [19662.734178] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19662.745672] Lustre: Skipped 1 previous similar message [19781.399916] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19866.517068] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19898.590439] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.117@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19900.277931] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19900.297745] LustreError: Skipped 1 previous similar message [19900.953207] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [19900.965462] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (21): c: 5, oc: 0, rc: 8 [19900.979228] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655030/real 1651655031] req@ffff99e7db728480 x1731865889557824/t0(0) o400->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/224 e 0 to 1 dl 1651655074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'' [19900.979233] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655014/real 1651655031] req@ffff99e8117a0d80 x1731865889050560/t0(0) o41->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651655058 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [19900.979239] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19900.979260] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [19900.979261] Lustre: Skipped 1 previous similar message [19902.608674] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19902.628490] LustreError: Skipped 1 previous similar message [19905.293654] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19905.313487] LustreError: Skipped 7 previous similar messages [19910.301836] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19910.321663] LustreError: Skipped 12 previous similar messages [19916.953971] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.109@o2ib: 1 seconds [19916.965340] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 51 previous similar messages [19917.879072] Lustre: 7037:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/-295), not sending early reply req@ffff99eb78c0da00 x1731763470852416/t0(0) o101->00629187-5c6d-4754-a616-e35b213076e9@192.168.1.121@o2ib:442/0 lens 1312/4072 e 1 to 0 dl 1651655052 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19917.910928] Lustre: 7037:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [19918.629183] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.117@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19918.649009] LustreError: Skipped 30 previous similar messages [19935.343042] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19935.362885] LustreError: Skipped 95 previous similar messages [19967.506691] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.102@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19967.526519] LustreError: Skipped 190 previous similar messages [19972.247610] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [20031.575005] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.135@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [20031.594857] LustreError: Skipped 381 previous similar messages [20079.857446] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20080.375714] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20080.386240] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20080.896518] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20080.907027] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.431028] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20081.441555] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.542903] sd 0:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [20081.553289] sd 0:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current] [20081.553304] sd 0:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553311] sd 0:0:0:0: [sdc] tag#2 Sense Key : Illegal Request [current] [20081.553319] sd 0:0:0:0: [sdc] tag#2 <>ASC=0x94 ASCQ=0x1 [20081.553325] sd 0:0:0:0: [sdc] tag#2 CDB: Write(16) 8a 00 00 00 00 00 40 00 00 48 00 00 00 08 00 00 [20081.553330] blk_update_request: I/O error, dev sdc, sector 1073741896 [20081.553350] sd 0:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553354] sd 0:0:0:0: [sdc] tag#3 Sense Key : Illegal Request [current] [20081.553359] sd 0:0:0:0: [sdc] tag#3 <>ASC=0x94 ASCQ=0x1 [20081.553363] sd 0:0:0:0: [sdc] tag#3 CDB: Write(16) 8a 00 00 00 00 00 40 40 00 50 00 00 00 08 00 00 [20081.553366] blk_update_request: I/O error, dev sdc, sector 1077936208 [20081.553376] sd 0:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553380] sd 0:0:0:0: [sdc] tag#4 Sense Key : Illegal Request [current] [20081.553385] sd 0:0:0:0: [sdc] tag#4 <>ASC=0x94 ASCQ=0x1 [20081.553388] sd 0:0:0:0: [sdc] tag#4 CDB: Write(16) 8a 00 00 00 00 00 40 40 00 80 00 00 00 08 00 00 [20081.553390] blk_update_request: I/O error, dev sdc, sector 1077936256 [20081.553400] sd 0:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553404] sd 0:0:0:0: [sdc] tag#5 Sense Key : Illegal Request [current] [20081.553408] sd 0:0:0:0: [sdc] tag#5 <>ASC=0x94 ASCQ=0x1 [20081.553412] sd 0:0:0:0: [sdc] tag#5 CDB: Write(16) 8a 00 00 00 00 00 40 00 00 80 00 00 00 08 00 00 [20081.553414] blk_update_request: I/O error, dev sdc, sector 1073741952 [20081.553423] sd 0:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553427] sd 0:0:0:0: [sdc] tag#6 Sense Key : Illegal Request [current] [20081.553432] sd 0:0:0:0: [sdc] tag#6 <>ASC=0x94 ASCQ=0x1 [20081.553435] sd 0:0:0:0: [sdc] tag#6 CDB: Write(16) 8a 00 00 00 00 00 40 00 03 c8 00 00 00 08 00 00 [20081.553437] blk_update_request: I/O error, dev sdc, sector 1073742792 [20081.553446] sd 0:0:0:0: [sdc] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553449] sd 0:0:0:0: [sdc] tag#7 Sense Key : Illegal Request [current] [20081.553454] sd 0:0:0:0: [sdc] tag#7 <>ASC=0x94 ASCQ=0x1 [20081.553458] sd 0:0:0:0: [sdc] tag#7 CDB: Write(16) 8a 00 00 00 00 00 3f c0 00 80 00 00 00 08 00 00 [20081.553459] blk_update_request: I/O error, dev sdc, sector 1069547648 [20081.553469] sd 0:0:0:0: [sdc] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553472] sd 0:0:0:0: [sdc] tag#8 Sense Key : Illegal Request [current] [20081.553477] sd 0:0:0:0: [sdc] tag#8 <>ASC=0x94 ASCQ=0x1 [20081.553481] sd 0:0:0:0: [sdc] tag#8 CDB: Write(16) 8a 00 00 00 00 00 3f c0 03 c8 00 00 00 08 00 00 [20081.553482] blk_update_request: I/O error, dev sdc, sector 1069548488 [20081.553491] sd 0:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553494] sd 0:0:0:0: [sdc] tag#9 Sense Key : Illegal Request [current] [20081.553499] sd 0:0:0:0: [sdc] tag#9 <>ASC=0x94 ASCQ=0x1 [20081.553503] sd 0:0:0:0: [sdc] tag#9 CDB: Write(16) 8a 00 00 00 00 00 3f c0 00 20 00 00 00 08 00 00 [20081.553504] blk_update_request: I/O error, dev sdc, sector 1069547552 [20081.553514] sd 0:0:0:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553517] sd 0:0:0:0: [sdc] tag#10 Sense Key : Illegal Request [current] [20081.553522] sd 0:0:0:0: [sdc] tag#10 <>ASC=0x94 ASCQ=0x1 [20081.553526] sd 0:0:0:0: [sdc] tag#10 CDB: Write(16) 8a 00 00 00 00 00 3f 2c 10 40 00 00 00 08 00 00 [20081.553528] blk_update_request: I/O error, dev sdc, sector 1059852352 [20081.553535] blk_update_request: I/O error, dev sdc, sector 1059852224 [20081.553846] device-mapper: multipath: Failing path 8:32. [20081.951288] sd 0:0:0:0: [sdc] tag#1 <>ASC=0x94 ASCQ=0x1 [20081.956794] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20081.956804] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.977827] sd 0:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 b1 23 78 00 00 00 08 00 00 [20082.290399] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [20082.299082] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20082.879213] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [20083.899568] Buffer I/O error on dev dm-1, logical block 873318384, async page read [20084.915892] Buffer I/O error on dev dm-1, logical block 0, async page read [20085.935954] Buffer I/O error on dev dm-1, logical block 873318399, async page read [20086.429416] device-mapper: multipath: Reinstating path 8:32. [20086.445146] Buffer I/O error on dev dm-1, logical block 0, async page read [20086.784054] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [20086.954225] scsi_io_completion: 127 callbacks suppressed [20086.960184] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20086.970483] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20086.978085] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20086.984798] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20086.994707] blk_update_request: 133 callbacks suppressed [20087.000636] blk_update_request: critical target error, dev sdh, sector 0 [20087.008139] blk_update_request: critical target error, dev dm-1, sector 0 [20087.015727] Buffer I/O error on dev dm-1, logical block 0, async page read [20087.525001] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20087.535334] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20087.542936] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20087.549659] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [20087.559577] blk_update_request: critical target error, dev sdh, sector 24 [20087.567226] blk_update_request: critical target error, dev dm-1, sector 24 [20087.574933] Buffer I/O error on dev dm-1, logical block 3, async page read [20088.094706] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20088.105030] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20088.112622] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20088.119349] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [20088.129258] blk_update_request: critical target error, dev sdh, sector 6986547072 [20088.137647] blk_update_request: critical target error, dev dm-1, sector 6986547072 [20088.647360] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20088.657698] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20088.665312] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20088.672034] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [20088.681946] blk_update_request: critical target error, dev sdh, sector 6986547072 [20088.690348] blk_update_request: critical target error, dev dm-1, sector 6986547072 [20088.698810] Buffer I/O error on dev dm-1, logical block 873318384, async page read [20089.214077] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20089.224399] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20089.231983] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20089.238706] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [20089.248630] blk_update_request: critical target error, dev sdh, sector 0 [20089.256141] blk_update_request: critical target error, dev dm-1, sector 0 [20089.765608] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20089.775930] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20089.783516] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20089.790233] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20089.800186] Buffer I/O error on dev dm-1, logical block 0, async page read [20090.308795] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20090.319113] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20090.326698] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20090.333422] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [20090.844084] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20090.854407] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20090.861999] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20090.868710] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [20090.878660] Buffer I/O error on dev dm-1, logical block 873318399, async page read [20091.388299] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20091.398621] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20091.406206] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20091.412923] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20091.422873] Buffer I/O error on dev dm-1, logical block 0, async page read [20091.927919] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [20091.937454] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20091.947766] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20091.955352] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20091.962068] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20091.972298] Buffer I/O error on dev dm-1, logical block 0, async page read [20092.484368] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20092.494685] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20092.502278] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20092.508988] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [20092.518895] blk_update_request: 10 callbacks suppressed [20092.524727] blk_update_request: critical target error, dev sdh, sector 24 [20092.532852] blk_update_request: critical target error, dev dm-1, sector 24 [20092.540584] Buffer I/O error on dev dm-1, logical block 3, async page read [20106.441599] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e818726000, cur 1651655237 expire 1651655087 last 1651655010 [20107.352696] Lustre: MGS: haven't heard from client 2d17adbe-08f3-468b-b9e3-3494dfdac5b3 (at 192.168.1.109@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e5b5523400, cur 1651655238 expire 1651655088 last 1651655010 [20118.769773] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [20159.738184] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.107@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [20159.758039] LustreError: Skipped 839 previous similar messages [20166.326630] LDISKFS-fs (dm-0): recovery complete [20166.332330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [20168.843465] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.120@o2ib (not set up) [20169.329832] Lustre: soaked-MDT0000: Received LWP connection from 0@lo, removing former export from 192.168.1.109@o2ib [20169.338519] Lustre: soaked-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [20169.342672] Lustre: soaked-MDT0001: in recovery but waiting for the first client to connect [20169.410362] Lustre: soaked-MDT0001: Will be in recovery for at least 2:30, or until 22 clients reconnect [20220.375840] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.108@o2ib (at 0@lo) [20220.377523] Lustre: soaked-MDT0001: Recovery over after 0:51, of 22 clients 22 recovered and 0 were evicted. [20221.905877] Lustre: Failing over soaked-MDT0001 [20221.996382] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.122@o2ib (stopping) [20222.008582] LustreError: 36531:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) soaked-OST0009-osc-MDT0001: cannot cleanup orphans: rc = -5 [20222.023267] LustreError: 36531:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [20224.004917] LustreError: 3512:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff99ebf580ad00 x1731865914476544/t0(0) o41->soaked-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [20224.029236] LustreError: 3512:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message [20224.433939] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20224.433942] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20224.433970] Lustre: Skipped 5 previous similar messages [20224.459449] Lustre: Skipped 1 previous similar message [20224.916983] LustreError: 3517:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff99eb47cdd100 x1731865914477440/t0(0) o41->soaked-MDT0003-osp-MDT0001@192.168.1.111@o2ib:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [20225.413215] LustreError: 11-0: soaked-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [20225.424418] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20226.923954] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.136@o2ib (stopping) [20226.933771] Lustre: Skipped 8 previous similar messages [20227.883710] LustreError: 36836:0:(ldlm_resource.c:1124:ldlm_resource_complain()) mdt-soaked-MDT0001_UUID: namespace resource [0x240002353:0x1cd4c:0x0].0x54cb7172 (ffff99ec6452dc80) refcount nonzero (1) after lock cleanup; forcing cleanup. [20228.193029] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20228.705026] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20228.717677] LustreError: Skipped 1 previous similar message [20229.724097] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20229.736743] LustreError: Skipped 3 previous similar messages [20231.743198] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20231.755838] LustreError: Skipped 7 previous similar messages [20231.938373] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.117@o2ib (stopping) [20231.948217] Lustre: Skipped 34 previous similar messages [20235.763389] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20235.776035] LustreError: Skipped 15 previous similar messages [20240.452653] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.110@o2ib (stopping) [20240.462519] Lustre: Skipped 65 previous similar messages [20243.782715] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20243.795371] LustreError: Skipped 31 previous similar messages [20256.873251] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.126@o2ib (stopping) [20256.883094] Lustre: Skipped 109 previous similar messages [20259.804443] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20259.817084] LustreError: Skipped 63 previous similar messages [20278.499861] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20278.522587] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) Skipped 1039 previous similar messages [20282.521971] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20282.544696] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) Skipped 959 previous similar messages [20289.569510] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20289.579338] Lustre: Skipped 248 previous similar messages [20290.986148] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20291.008877] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 1139 previous similar messages [20291.826893] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20291.839540] LustreError: Skipped 127 previous similar messages [20320.719349] LustreError: 3776:0:(lod_qos.c:115:lod_statfs_and_check()) soaked-MDT0001-mdtlov: statfs: rc = -108 [20320.731100] LustreError: 3776:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff99ec4190e800 ns: mdt-soaked-MDT0001_UUID lock: ffff99e798808d80/0x322afadd04c8f0d3 lrc: 3/0,0 mode: CR/CR res: [0x24000235e:0x5575:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.1.136@o2ib remote: 0x527c91d1be39cad9 expref: 3 pid: 3776 timeout: 0 lvb_type: 0 [20320.770586] Lustre: 3776:0:(service.c:2327:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (13/87s); client may timeout req@ffff99e7998f1f80 x1731763240218368/t25769805658(0) o101->d85fd948-cdf6-4a2e-8806-76ec090bacdc@192.168.1.136@o2ib:754/0 lens 1304/600 e 0 to 0 dl 1651655364 ref 1 fl Complete:/0/0 rc -19/-19 job:'' [20321.441237] LustreError: 3564:0:(lod_qos.c:115:lod_statfs_and_check()) soaked-MDT0001-mdtlov: statfs: rc = -108 [20321.452526] LustreError: 3564:0:(lod_qos.c:115:lod_statfs_and_check()) Skipped 83 previous similar messages [20321.463706] LustreError: 3564:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff99ec4190e400 ns: mdt-soaked-MDT0001_UUID lock: ffff99ebe85ea640/0x322afadd04c9c397 lrc: 3/0,0 mode: CR/CR res: [0x24000234d:0x11022:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.1.137@o2ib remote: 0x6ba7b49558674dbf expref: 3 pid: 3564 timeout: 0 lvb_type: 0 [20321.503200] LustreError: 3564:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [20321.514435] Lustre: 3564:0:(service.c:2327:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/81s); client may timeout req@ffff99ec3c9cc800 x1731763737640064/t25769805661(0) o101->cb2a3a98-2672-400e-b628-aaad60526b53@192.168.1.137@o2ib:6/0 lens 1304/600 e 0 to 0 dl 1651655371 ref 1 fl Complete:/0/0 rc -19/-19 job:'' [20332.860212] Lustre: server umount soaked-MDT0001 complete [20402.199412] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [20433.977365] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 18 seconds [20433.989612] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.111@o2ib (20): c: 7, oc: 0, rc: 8 [20434.003360] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655545/real 1651655564] req@ffff99e7bb524800 x1731865920634304/t0(0) o41->soaked-MDT0003-osp-MDT0000@192.168.1.111@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651655589 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [20434.036501] Lustre: soaked-MDT0003-osp-MDT0000: Connection to soaked-MDT0003 (at 192.168.1.111@o2ib) was lost; in progress operations using this service will wait for recovery to complete [20437.977540] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.111@o2ib: 6 seconds [20437.988903] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 7 previous similar messages [20627.423556] Lustre: MGS: haven't heard from client c4de137e-ae46-4871-918a-a490e9e834d3 (at 192.168.1.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e5b54db800, cur 1651655758 expire 1651655608 last 1651655531 [20640.574836] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e818726800, cur 1651655771 expire 1651655621 last 1651655544 [20740.630155] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.111@o2ib, keep former export from same NID [20811.500672] Lustre: soaked-MDT0003-osp-MDT0000: Connection restored to 192.168.1.111@o2ib (at 192.168.1.111@o2ib) [21339.418695] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21339.441443] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) Skipped 1599 previous similar messages [21341.847396] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21341.870195] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) Skipped 179 previous similar messages [21345.854415] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21345.877170] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) Skipped 459 previous similar messages [21353.912541] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21353.935274] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) Skipped 959 previous similar messages [21370.059057] LustreError: 34266:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21370.081913] LustreError: 34266:0:(out_handler.c:910:out_tx_end()) Skipped 719 previous similar messages [22137.376555] LustreError: 5290:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99ec644d6400 [22137.387546] LustreError: 5290:0:(llog_cat.c:604:llog_cat_add_rec()) Skipped 2 previous similar messages [22137.398084] LustreError: 5290:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0003-osp-MDT0000: write updates failed: rc = -116 [22137.411209] LustreError: 5290:0:(update_trans.c:1062:top_trans_stop()) Skipped 2 previous similar messages [22138.279594] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [22138.293980] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1235 previous similar messages [22138.305466] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [22138.320331] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1235 previous similar messages [22157.231150] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [22157.245541] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 519 previous similar messages [22157.256953] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [22157.271820] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 519 previous similar messages [22446.657561] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657570/real 0] req@ffff99e7a140e780 x1731866015376256/t0(0) o13->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657577 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22446.689230] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22447.618612] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657571/real 0] req@ffff99e7a140ec00 x1731866015436992/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657578 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22447.650272] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22449.924750] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657573/real 0] req@ffff99ec3cc3cc80 x1731866015509312/t0(0) o13->soaked-OST000a-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657580 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22449.925723] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22449.975086] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [22456.068997] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [22456.081253] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (18): c: 0, oc: 0, rc: 8 [22662.392489] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e8119c5000, cur 1651657793 expire 1651657643 last 1651657566 [22663.896838] Lustre: MGS: haven't heard from client 4c64ca59-6fb9-4c18-a09c-2e3212225e10 (at 192.168.1.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99ec41f2f000, cur 1651657794 expire 1651657644 last 1651657566 [22663.920554] Lustre: Skipped 3 previous similar messages [22738.726954] perf: interrupt took too long (3398 > 3255), lowering kernel.perf_event_max_sample_rate to 58000 [22776.083497] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [22776.094880] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [22844.086578] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [22844.097948] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 20 previous similar messages [22976.092605] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [22976.103999] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 48 previous similar messages [23054.591961] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23195.172669] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23245.998801] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23303.759245] LustreError: 11-0: soaked-OST0002-osc-MDT0000: operation ost_destroy to node 192.168.1.107@o2ib failed: rc = -19 [23303.759462] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23303.759465] Lustre: Skipped 1 previous similar message [23303.796192] LustreError: Skipped 3 previous similar messages [23303.934052] LustreError: 3608:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST0002-osc-MDT0000: can't precreate: rc = -19 [23303.947019] LustreError: 3608:0:(osp_precreate.c:1339:osp_precreate_thread()) soaked-OST0002-osc-MDT0000: cannot precreate objects: rc = -19 [23321.137063] LustreError: 11-0: soaked-OST000a-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23321.149611] LustreError: Skipped 1 previous similar message [23321.155876] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23322.657135] LustreError: 11-0: soaked-OST0006-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23322.669744] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23328.338954] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [23328.350453] Lustre: Skipped 1 previous similar message [23330.401476] LustreError: 11-0: soaked-OST000e-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23330.414079] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23697.284927] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23697.299324] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 307 previous similar messages [23697.310733] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23697.325593] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 307 previous similar messages [23702.125552] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23702.139944] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 36 previous similar messages [23702.151245] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23702.166105] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 36 previous similar messages [23714.425816] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23714.440231] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 46 previous similar messages [23714.451522] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23714.466378] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 46 previous similar messages [23735.925214] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [23735.939446] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 205 previous similar messages [23735.950837] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [23735.965543] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 205 previous similar messages [23775.722026] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [23775.736237] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 400 previous similar messages [23775.747620] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [23775.762301] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 400 previous similar messages [24584.011761] Lustre: 3599:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651659702/real 0] req@ffff99eb79a46300 x1731866074594816/t0(0) o1000->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:24/4 lens 488/4320 e 0 to 1 dl 1651659714 ref 3 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [24584.043826] Lustre: soaked-MDT0002-osp-MDT0000: Connection to soaked-MDT0002 (at 192.168.1.110@o2ib) was lost; in progress operations using this service will wait for recovery to complete [24584.740796] Lustre: 3562:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651659703/real 0] req@ffff99e799e9cc80 x1731866074600832/t0(0) o101->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:24/4 lens 328/344 e 0 to 1 dl 1651659715 ref 3 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [24584.772647] Lustre: 3562:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [24591.162074] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: tx_queue(WSQ:001), 19 seconds [24591.174131] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.110@o2ib (24): c: 0, oc: 0, rc: 8 [24591.187909] Lustre: 3502:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659702/real 1651659721] req@ffff99e822bc5100 x1731866074597376/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659749 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24595.162262] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 7 seconds [24595.173618] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [24599.162504] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659707/real 1651659729] req@ffff99ebf7d9d100 x1731866074612608/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659754 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24599.195812] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [24603.162673] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659707/real 1651659733] req@ffff99ec3e1e3f00 x1731866074614272/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659754 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24603.195975] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [24627.163621] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 0 seconds [24627.174985] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 27 previous similar messages [24703.166904] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 2 seconds [24703.178278] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [24772.878865] ptlrpc_watchdog_fire: 3 callbacks suppressed [24772.878881] Lustre: mdt00_007: service thread pid 3780 was inactive for 200.722 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [24772.878884] Lustre: Skipped 3 previous similar messages [24772.906970] Lustre: mdt00_010: service thread pid 5287 was inactive for 200.132 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [24772.927854] Pid: 5287, comm: mdt00_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [24772.938835] Call Trace: [24772.941667] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [24772.947466] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [24772.953372] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [24772.959241] [<0>] osp_md_object_lock+0x160/0x300 [osp] [24772.965019] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [24772.970498] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [24772.975813] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [24772.981963] [<0>] mdt_reint_unlink+0x6d5/0x1b80 [mdt] [24772.987629] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [24772.992810] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [24772.998563] [<0>] mdt_reint+0x67/0x150 [mdt] [24773.003389] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [24773.009584] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [24773.016625] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [24773.022078] [<0>] kthread+0xd1/0xe0 [24773.025978] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [24773.031758] [<0>] 0xfffffffffffffffe [24788.598118] Lustre: MGS: haven't heard from client fe865eff-e137-4afe-a54d-f44c908c3b80 (at 192.168.1.110@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e5b54dd800, cur 1651659919 expire 1651659769 last 1651659692 [24795.607991] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0002-mdtlov_UUID (at 192.168.1.110@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e818726400, cur 1651659926 expire 1651659776 last 1651659697 [25050.142889] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.110@o2ib, keep former export from same NID [25116.355826] Lustre: soaked-MDT0002-osp-MDT0000: Connection restored to 192.168.1.110@o2ib (at 192.168.1.110@o2ib) [25116.367317] Lustre: Skipped 3 previous similar messages [26562.219812] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26562.234210] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 15 previous similar messages [26562.245508] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26562.260375] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 15 previous similar messages [26573.478456] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26573.492866] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 99 previous similar messages [26573.504153] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26573.519021] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 99 previous similar messages [26593.502229] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26593.516630] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 203 previous similar messages [26593.528007] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26593.542897] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 203 previous similar messages [26631.365225] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26631.379626] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 288 previous similar messages [26631.391028] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26631.405900] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 288 previous similar messages [26635.159877] Lustre: 3499:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651661758/real 0] req@ffff99ec41c99200 x1731866138744320/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651661765 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [26635.191530] Lustre: 3499:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [26635.202385] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [26635.876892] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [26635.895558] Lustre: Skipped 2 previous similar messages [26644.251267] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [26644.263527] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (20): c: 0, oc: 0, rc: 8 [26644.277249] Lustre: 3491:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651661756/real 1651661774] req@ffff99e7ca7a2d00 x1731866138588672/t0(0) o400->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:28/4 lens 224/224 e 0 to 1 dl 1651661800 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'' [26644.277724] LustreError: 3632:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST000e-osc-MDT0000: can't precreate: rc = -5 [26644.323385] Lustre: 3491:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [26836.232335] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80db19000, cur 1651661966 expire 1651661816 last 1651661739 [26842.778052] Lustre: MGS: haven't heard from client 260df422-b357-49b9-b86c-e462e7402cb7 (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80f28d000, cur 1651661973 expire 1651661823 last 1651661746 [26842.801776] Lustre: Skipped 3 previous similar messages [26952.264781] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [26952.276146] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [26968.265451] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 7 seconds [26968.276825] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [27000.266849] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 10 seconds [27000.278332] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 11 previous similar messages [27064.269695] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 36 seconds [27064.281167] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 29 previous similar messages [27522.513349] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27557.319037] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27602.603420] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27647.651494] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [29638.927715] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [29638.950452] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 1379 previous similar messages [30046.955568] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665170/real 0] req@ffff99e7a2ac7980 x1731866190919168/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665177 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30046.987259] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30048.037621] Lustre: 3502:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665171/real 0] req@ffff99e7ca7a3a80 x1731866191046080/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665178 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30048.069297] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30050.985748] Lustre: 3521:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665172/real 0] req@ffff99ec3d7e1680 x1731866191099072/t0(0) o13->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665179 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30050.985753] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665171/real 0] req@ffff99ebef1e5580 x1731866191007168/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665178 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30050.985775] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30056.401033] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [30056.413295] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (19): c: 0, oc: 0, rc: 8 [30056.427197] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651665171/real 1651665186] req@ffff99e7a1106780 x1731866191028416/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651665258 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [30056.460217] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [30243.056606] Lustre: MGS: haven't heard from client 4ae9cc45-77ab-4a37-98e1-c34a7d3e0402 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e813b12000, cur 1651665373 expire 1651665223 last 1651665145 [30324.412772] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 1 seconds [30324.424131] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 61 previous similar messages [30340.413485] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 12 seconds [30340.424952] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 11 previous similar messages [30372.414897] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 44 seconds [30372.426361] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 15 previous similar messages [30436.417709] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 108 seconds [30436.429278] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [30699.393960] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30710.046931] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30757.501971] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30803.277455] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [31695.412547] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31695.435284] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 31 previous similar messages [31696.244123] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31696.266843] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 83 previous similar messages [31697.316034] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31697.338775] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 35 previous similar messages [31699.387208] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31699.409956] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) Skipped 219 previous similar messages [31707.957541] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31707.980308] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 319 previous similar messages [31716.206641] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31716.229370] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 1299 previous similar messages [31732.263752] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31732.286471] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 1179 previous similar messages [31931.407319] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [31931.421712] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 235 previous similar messages [31931.433093] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [31931.447971] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 235 previous similar messages [31941.143574] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [31941.157777] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 89 previous similar messages [31941.169068] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [31941.183751] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 89 previous similar messages [31961.222111] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [31961.236295] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 262 previous similar messages [31961.247665] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [31961.262326] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 262 previous similar messages [32001.067231] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [32001.081445] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 586 previous similar messages [32001.092850] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [32001.107517] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 586 previous similar messages [32879.391297] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32879.414033] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 999 previous similar messages [32883.447814] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32883.470569] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 359 previous similar messages [32891.493754] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32891.516482] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) Skipped 1419 previous similar messages [32907.494860] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32907.517591] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 1659 previous similar messages [33891.195764] perf: interrupt took too long (4271 > 4247), lowering kernel.perf_event_max_sample_rate to 46000 [33981.104631] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33981.127357] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 699 previous similar messages [33985.389464] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33985.412204] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 667 previous similar messages [33994.813534] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33994.836280] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 991 previous similar messages [34103.193256] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34103.207660] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 394 previous similar messages [34103.219050] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34103.233911] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 394 previous similar messages [34112.688305] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34112.702703] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 128 previous similar messages [34112.714107] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34112.728975] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 128 previous similar messages [34131.471483] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34131.485869] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 269 previous similar messages [34131.497243] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34131.512106] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 269 previous similar messages [34174.377080] Lustre: 3503:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651669297/real 0] req@ffff99e7a2d60000 x1731866385351744/t0(0) o103->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651669304 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [34174.409073] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [34174.427735] Lustre: Skipped 1 previous similar message [34177.219180] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651669300/real 0] req@ffff99e7a1e0cc80 x1731866385354560/t0(0) o400->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/224 e 0 to 1 dl 1651669307 ref 2 fl Rpc:XNr/0/ffffffff rc 0/-1 job:'' [34177.251138] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [34185.083471] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.125@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34185.103287] LustreError: Skipped 191 previous similar messages [34185.583489] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 18 seconds [34185.595732] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (18): c: 2, oc: 0, rc: 8 [34217.106934] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.120@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34217.126761] LustreError: Skipped 150 previous similar messages [34281.779358] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34281.799186] LustreError: Skipped 390 previous similar messages [34352.022230] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34352.535871] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34352.546393] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34353.059581] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34353.070099] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34353.579957] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34353.590499] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34354.100502] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34354.111017] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34354.624230] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34354.634753] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34355.150202] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34355.160718] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34355.676804] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34355.687332] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34356.210282] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34356.220797] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34356.332611] sd 0:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [34356.342940] sd 0:0:0:0: [sdc] tag#0 Sense Key : Illegal Request [current] [34356.350632] sd 0:0:0:0: [sdc] tag#0 <>ASC=0x94 ASCQ=0x1 [34356.357347] sd 0:0:0:0: [sdc] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 a0 45 e0 00 00 00 f8 00 00 [34356.367360] blk_update_request: I/O error, dev sdc, sector 10503648 [34356.374405] device-mapper: multipath: Failing path 8:32. [34356.500803] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [34357.016524] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34357.026858] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34357.034451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34357.041168] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34357.051096] blk_update_request: critical target error, dev sdh, sector 6986547072 [34357.059491] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34357.071164] device-mapper: multipath: Failing path 8:32. [34357.593607] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34357.603907] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34357.611504] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34357.618222] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34357.628132] blk_update_request: critical target error, dev sdh, sector 6986547072 [34357.636542] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34357.645033] Buffer I/O error on dev dm-1, logical block 873318384, async page read [34358.160350] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34358.170669] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34358.178257] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34358.184983] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [34358.194908] blk_update_request: critical target error, dev sdh, sector 0 [34358.202434] blk_update_request: critical target error, dev dm-1, sector 0 [34358.718829] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34358.729141] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34358.736745] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34358.743466] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34358.753379] blk_update_request: critical target error, dev sdh, sector 0 [34358.760897] blk_update_request: critical target error, dev dm-1, sector 0 [34358.768501] Buffer I/O error on dev dm-1, logical block 0, async page read [34359.277144] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34359.287463] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34359.295057] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34359.301769] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34359.311687] blk_update_request: critical target error, dev sdh, sector 6986547192 [34359.827163] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34359.837486] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34359.845079] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34359.851792] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34359.861741] Buffer I/O error on dev dm-1, logical block 873318399, async page read [34360.377194] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34360.387515] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34360.395110] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34360.401834] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34360.411784] Buffer I/O error on dev dm-1, logical block 0, async page read [34360.922243] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34360.932561] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34360.940148] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34360.946858] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34360.956790] Buffer I/O error on dev dm-1, logical block 0, async page read [34361.078071] device-mapper: multipath: Reinstating path 8:32. [34361.129878] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [34361.466169] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34361.476486] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34361.484079] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34361.490795] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [34361.500710] blk_update_request: 7 callbacks suppressed [34361.506453] blk_update_request: critical target error, dev sdh, sector 24 [34361.514075] blk_update_request: critical target error, dev dm-1, sector 24 [34361.521791] Buffer I/O error on dev dm-1, logical block 3, async page read [34362.042252] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34362.052599] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34362.060196] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34362.066905] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34362.076822] blk_update_request: critical target error, dev sdh, sector 6986547072 [34362.085230] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34362.503808] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [34362.595955] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34362.606270] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34362.613874] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34362.620589] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34362.630506] blk_update_request: critical target error, dev sdh, sector 6986547072 [34362.638926] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34362.647412] Buffer I/O error on dev dm-1, logical block 873318384, async page read [34363.170379] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34363.180699] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34363.188285] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34363.194998] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [34363.204908] blk_update_request: critical target error, dev sdh, sector 0 [34363.212430] blk_update_request: critical target error, dev dm-1, sector 0 [34363.727299] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34363.737607] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34363.745200] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34363.751912] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34363.761821] blk_update_request: critical target error, dev sdh, sector 0 [34363.770327] blk_update_request: critical target error, dev dm-1, sector 0 [34363.777923] Buffer I/O error on dev dm-1, logical block 0, async page read [34364.293970] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34364.304284] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34364.311869] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34364.318591] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34364.829346] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34364.839649] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34364.847242] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34364.853953] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34364.863889] Buffer I/O error on dev dm-1, logical block 873318399, async page read [34365.377421] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34365.387752] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34365.395344] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34365.402054] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34365.412403] Buffer I/O error on dev dm-1, logical block 0, async page read [34365.927407] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34365.937722] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34365.945326] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34365.952047] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34365.961991] Buffer I/O error on dev dm-1, logical block 0, async page read [34366.477501] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34366.487810] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34366.495406] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34366.502117] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [34366.512019] blk_update_request: 8 callbacks suppressed [34366.517771] blk_update_request: critical target error, dev sdh, sector 24 [34366.525383] blk_update_request: critical target error, dev dm-1, sector 24 [34366.533074] Buffer I/O error on dev dm-1, logical block 3, async page read [34368.179574] Lustre: mdt01_002: service thread pid 3565 was inactive for 200.635 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [34368.200472] Pid: 3565, comm: mdt01_002 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [34368.211474] Call Trace: [34368.214349] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [34368.220162] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [34368.226079] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [34368.231943] [<0>] osp_md_object_lock+0x160/0x300 [osp] [34368.237729] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [34368.243205] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [34368.248510] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [34368.254670] [<0>] mdt_reint_unlink+0x6d5/0x1b80 [mdt] [34368.260334] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [34368.265533] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [34368.271298] [<0>] mdt_reint+0x67/0x150 [mdt] [34368.276152] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [34368.282351] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [34368.289417] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [34368.294876] [<0>] kthread+0xd1/0xe0 [34368.298781] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [34368.304550] [<0>] 0xfffffffffffffffe [34389.509972] Lustre: MGS: haven't heard from client b7526d44-6fcc-4133-a928-8ab0781133c5 (at 192.168.1.109@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff99e7a165f000, cur 1651669519 expire 1651669369 last 1651669289 [34389.533684] Lustre: Skipped 4 previous similar messages [34398.388859] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [34409.865775] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.123@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34409.885592] LustreError: Skipped 816 previous similar messages [34440.979962] LDISKFS-fs (dm-0): recovery complete [34440.985672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [34442.715058] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.104@o2ib (not set up) [34442.725095] Lustre: Skipped 316 previous similar messages [34442.815341] Lustre: soaked-MDT0000: Received MDS connection from 0@lo, removing former export from 192.168.1.109@o2ib [34442.973240] Lustre: soaked-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [34442.993193] Lustre: soaked-MDT0001: in recovery but waiting for the first client to connect [34443.000237] Lustre: soaked-MDT0001: Will be in recovery for at least 2:30, or until 22 clients reconnect [34448.203256] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [34448.217492] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 533 previous similar messages [34448.228891] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [34448.243579] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 533 previous similar messages [34458.152653] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.108@o2ib (at 0@lo) [34458.153363] Lustre: soaked-MDT0001: Recovery over after 0:15, of 22 clients 22 recovered and 0 were evicted. [34460.002987] Lustre: Failing over soaked-MDT0001 [34460.035786] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.110@o2ib (stopping) [34460.045605] Lustre: Skipped 4 previous similar messages [34460.052751] LustreError: 60248:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST0001-osc-MDT0001: can't precreate: rc = -5 [34460.053708] LustreError: 60250:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) soaked-OST0002-osc-MDT0001: cannot cleanup orphans: rc = -5 [34460.056380] LustreError: 60256:0:(osp_precreate.c:1339:osp_precreate_thread()) soaked-OST0005-osc-MDT0001: cannot precreate objects: rc = -5 [34460.094414] LustreError: 60248:0:(osp_precreate.c:676:osp_precreate_send()) Skipped 1 previous similar message [34460.205679] Lustre: server umount soaked-MDT0001 complete [34463.192063] LustreError: 11-0: soaked-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [34463.203318] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [34489.235172] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [34528.533647] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34528.548035] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 189 previous similar messages [34528.559413] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34528.574294] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 189 previous similar messages [34858.612152] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [34858.624410] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.111@o2ib (19): c: 0, oc: 0, rc: 8 [34858.638231] Lustre: 3560:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651669972/real 1651669988] req@ffff99e7a173b600 x1731866403973888/t0(0) o101->soaked-MDT0003-osp-MDT0000@192.168.1.111@o2ib:24/4 lens 328/344 e 0 to 1 dl 1651670018 ref 2 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [34858.638285] Lustre: soaked-MDT0003-osp-MDT0000: Connection to soaked-MDT0003 (at 192.168.1.111@o2ib) was lost; in progress operations using this service will wait for recovery to complete [34858.690109] Lustre: 3560:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [35035.856950] Lustre: mdt00_003: service thread pid 3654 was inactive for 200.354 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [35035.856953] Lustre: mdt00_011: service thread pid 5288 was inactive for 200.350 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35035.856956] Pid: 3561, comm: mdt00_001 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35035.856959] Lustre: Skipped 11 previous similar messages [35035.856960] Call Trace: [35035.857066] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35035.857130] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35035.857190] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35035.857208] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35035.857232] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35035.857253] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35035.857286] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35035.857312] [<0>] mdt_create+0xc39/0xe40 [mdt] [35035.857336] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35035.857361] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35035.857382] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35035.857404] [<0>] mdt_reint+0x67/0x150 [mdt] [35035.857487] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35035.857557] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35035.857625] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35035.857631] [<0>] kthread+0xd1/0xe0 [35035.857637] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35035.857667] [<0>] 0xfffffffffffffffe [35035.857671] Pid: 3781, comm: mdt00_008 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35035.857672] Call Trace: [35035.857762] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35035.857827] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35035.857887] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35035.857915] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35035.857936] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35035.857953] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35035.857981] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35035.858006] [<0>] mdt_create+0xc39/0xe40 [mdt] [35035.858030] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35035.858055] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35035.858076] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35035.858098] [<0>] mdt_reint+0x67/0x150 [mdt] [35035.858177] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35035.858248] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35035.858316] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35035.858320] [<0>] kthread+0xd1/0xe0 [35035.858324] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35035.858336] [<0>] 0xfffffffffffffffe [35036.124077] Lustre: Skipped 2 previous similar messages [35036.129922] Pid: 3654, comm: mdt00_003 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35036.140908] Call Trace: [35036.143699] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35036.149480] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35036.155355] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35036.161210] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35036.166971] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35036.172436] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35036.177715] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35036.183863] [<0>] mdt_create+0xc39/0xe40 [mdt] [35036.188849] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35036.194423] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35036.199593] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35036.205340] [<0>] mdt_reint+0x67/0x150 [mdt] [35036.210157] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35036.216329] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35036.223361] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35036.228816] [<0>] kthread+0xd1/0xe0 [35036.232717] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35036.238465] [<0>] 0xfffffffffffffffe [35051.427856] Lustre: MGS: haven't heard from client 0eb36178-2f47-447e-9910-a57e14c3ddf8 (at 192.168.1.111@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff99e80f15d000, cur 1651670181 expire 1651670031 last 1651669951 [35076.306738] Lustre: mdt00_006: service thread pid 3776 was inactive for 200.350 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35076.323061] Lustre: Skipped 2 previous similar messages [35175.961125] LustreError: 3776:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651670006, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99e7a2263840/0x322afadd0daeda29 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1:0x0].0x0 bits 0x13/0x0 rrc: 17 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 3776 timeout: 0 lvb_type: 0 [35176.005968] LustreError: dumping log to /tmp/lustre-log.1651670306.3776 [35371.690405] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.111@o2ib, keep former export from same NID [35429.090235] ptlrpc_watchdog_fire: 4 callbacks suppressed [35429.090244] Lustre: mdt01_004: service thread pid 3769 was inactive for 593.566 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35429.090246] Lustre: Skipped 4 previous similar messages [35429.118327] Lustre: mdt01_027: service thread pid 35261 was inactive for 593.572 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [35429.139312] Pid: 35261, comm: mdt01_027 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35429.150403] Call Trace: [35429.153266] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35429.159064] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35429.164961] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35429.170820] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35429.176611] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35429.182093] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35429.187385] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35429.193553] [<0>] mdt_create+0xc39/0xe40 [mdt] [35429.198537] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35429.204112] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35429.209304] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35429.215063] [<0>] mdt_reint+0x67/0x150 [mdt] [35429.219928] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35429.226113] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35429.233153] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35429.238629] [<0>] kthread+0xd1/0xe0 [35429.242532] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35429.248309] [<0>] 0xfffffffffffffffe [35430.338332] Lustre: 22105:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99e769916300 x1731763694372608/t42961211840(0) o36->24816f25-f323-4fbc-a2f1-9bdf8a245040@192.168.1.128@o2ib:100/0 lens 520/448 e 22 to 0 dl 1651670565 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [35430.370759] Lustre: 22105:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages [35433.496816] Lustre: soaked-MDT0003-osp-MDT0000: Connection restored to 192.168.1.111@o2ib (at 192.168.1.111@o2ib) [35740.376380] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35740.399111] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) Skipped 2479 previous similar messages [35742.408194] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35742.430969] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 27 previous similar messages [35756.046954] LustreError: 3768:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35756.069694] LustreError: 3768:0:(out_handler.c:910:out_tx_end()) Skipped 23 previous similar messages [35764.307386] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35764.330128] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 19 previous similar messages [35781.120573] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35781.143317] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 47 previous similar messages [35813.697516] LustreError: 34307:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35813.720359] LustreError: 34307:0:(out_handler.c:910:out_tx_end()) Skipped 119 previous similar messages [36944.534477] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [36944.548693] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 366 previous similar messages [36944.560078] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [36944.574771] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 366 previous similar messages [36963.471657] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [36963.486072] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 156 previous similar messages [36963.497464] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [36963.512332] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 156 previous similar messages [37002.806012] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [37002.820440] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 424 previous similar messages [37002.831826] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [37002.846703] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 424 previous similar messages [37049.763738] LustreError: 3767:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99ec5d353000 [37049.774746] LustreError: 3767:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [37054.156407] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37054.179143] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) Skipped 75 previous similar messages [37065.402854] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37065.425591] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 7 previous similar messages [37087.198523] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37087.221263] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 215 previous similar messages [37120.009273] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37120.032003] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) Skipped 391 previous similar messages [37289.526468] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [37289.540857] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [37289.552371] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [37289.567229] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [37293.106056] LustreError: 22106:0:(llog_osd.c:626:llog_osd_write_rec()) soaked-MDT0001-osp-MDT0000: index 45193 already set in llog bitmap [0x2400032e9:0xee22:0x0] [37293.122294] LustreError: 22106:0:(llog_osd.c:628:llog_osd_write_rec()) LBUG [37293.130105] Pid: 22106, comm: mdt01_020 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37293.141187] Call Trace: [37293.143960] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [37293.149715] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [37293.155149] [<0>] llog_osd_write_rec+0x1a3a/0x1a70 [obdclass] [37293.161588] [<0>] llog_write_rec+0x293/0x590 [obdclass] [37293.167480] [<0>] llog_cat_add_rec+0x1d9/0x980 [obdclass] [37293.173543] [<0>] llog_add+0x182/0x1f0 [obdclass] [37293.178911] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [37293.184929] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [37293.190583] [<0>] lod_trans_stop+0x25c/0x340 [lod] [37293.195961] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [37293.201230] [<0>] mdd_create+0x154a/0x1cd0 [mdd] [37293.206418] [<0>] mdo_create+0x46/0x48 [mdt] [37293.211215] [<0>] mdt_create+0xab1/0xe40 [mdt] [37293.216199] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37293.221766] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37293.226942] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37293.232702] [<0>] mdt_reint+0x67/0x150 [mdt] [37293.237562] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37293.243740] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37293.250788] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37293.256246] [<0>] kthread+0xd1/0xe0 [37293.260154] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37293.265921] [<0>] 0xfffffffffffffffe [37293.270065] LustreError: dumping log to /tmp/lustre-log.1651672423.22106 [37493.565331] Lustre: mdt01_002: service thread pid 3565 was inactive for 200.486 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [37493.565341] Lustre: mdt01_020: service thread pid 22106 was inactive for 200.486 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37493.565345] Lustre: mdt00_001: service thread pid 3561 was inactive for 200.486 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37493.565350] Pid: 22105, comm: mdt00_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.565353] Lustre: Skipped 7 previous similar messages [37493.565356] Lustre: Skipped 9 previous similar messages [37493.565358] Call Trace: [37493.565467] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.565527] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.565585] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.565603] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.565628] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.565648] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.565693] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.565714] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.565748] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.565774] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.565799] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.565824] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.565845] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.565867] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.565951] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.566022] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.566089] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.566095] [<0>] kthread+0xd1/0xe0 [37493.566101] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37493.566133] [<0>] 0xfffffffffffffffe [37493.566136] Pid: 8311, comm: mdt00_014 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.566137] Call Trace: [37493.566217] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.566276] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.566334] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.566363] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.566383] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.566400] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.566424] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.566446] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.566467] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.566493] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.566518] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.566543] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.566563] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.566586] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.566664] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.566734] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.566801] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.566805] [<0>] kthread+0xd1/0xe0 [37493.566810] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37493.566821] [<0>] 0xfffffffffffffffe [37493.880610] Lustre: Skipped 2 previous similar messages [37493.886446] Pid: 3565, comm: mdt01_002 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.897439] Call Trace: [37493.900259] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.906425] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.912775] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.918630] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.924392] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.929777] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.935073] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.941599] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.947968] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.953234] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.958232] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.963801] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.968971] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.974726] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.979543] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.985710] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.992754] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.998206] [<0>] kthread+0xd1/0xe0 [37494.002141] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37494.007897] [<0>] 0xfffffffffffffffe [37570.368756] Lustre: mdt01_006: service thread pid 3774 was inactive for 200.042 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37593.083732] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [37593.102425] LustreError: 8311:0:(ldlm_request.c:142:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651672423, 300s ago), entering recovery for soaked-MDT0001_UUID@192.168.1.109@o2ib ns: soaked-MDT0001-osp-MDT0000 lock: ffff99ec3fc2ba80/0x322afadd0e9cc1b7 lrc: 4/0,1 mode: --/EX res: [0x2400032e9:0x182bf:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT gid 0 flags: 0x1000001000000 nid: local remote: 0xe4a46eacd6424a03 expref: -99 pid: 8311 timeout: 0 lvb_type: 0 [37593.115183] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [37593.160264] LustreError: 8311:0:(ldlm_request.c:142:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [37642.387933] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37642.410665] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 359 previous similar messages [37888.094802] Lustre: 22102:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99e815f05e80 x1731763812062080/t0(0) o36->ab8832f6-98e1-4d13-84e2-130e4afa61aa@192.168.1.138@o2ib:293/0 lens 512/448 e 23 to 0 dl 1651673023 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [37888.126252] Lustre: 22102:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [37894.149486] Lustre: soaked-MDT0000: Client a6431ad6-ad1a-403b-b2a6-2dbd7e58ec7b (at 192.168.1.120@o2ib) reconnecting [37894.161268] Lustre: Skipped 9 previous similar messages [37924.293443] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [37924.305715] Lustre: Skipped 1 previous similar message [37955.409744] ptlrpc_watchdog_fire: 6 callbacks suppressed [37955.415720] Lustre: mdt_rdpg01_008: service thread pid 3851 was inactive for 200.490 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [37955.437078] Pid: 3851, comm: mdt_rdpg01_008 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37955.448544] Call Trace: [37955.451310] [<0>] call_rwsem_down_write_failed+0x17/0x30 [37955.457295] [<0>] llog_cat_add_rec+0x11d/0x980 [obdclass] [37955.463382] [<0>] llog_add+0x182/0x1f0 [obdclass] [37955.468737] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [37955.474786] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [37955.480443] [<0>] lod_trans_stop+0x25c/0x340 [lod] [37955.485844] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [37955.491112] [<0>] mdd_attr_set+0x81c/0x1070 [mdd] [37955.496419] [<0>] mdt_mfd_close+0x771/0xbb0 [mdt] [37955.501694] [<0>] mdt_close_internal+0x141/0x240 [mdt] [37955.507452] [<0>] mdt_close+0x291/0x900 [mdt] [37955.512378] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37955.518556] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37955.525598] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37955.531054] [<0>] kthread+0xd1/0xe0 [37955.534961] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37955.540744] [<0>] 0xfffffffffffffffe [37965.394235] Lustre: 3787:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb64244380 x1731770926374272/t42961857331(0) o36->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:370/0 lens 552/448 e 5 to 0 dl 1651673100 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [37965.426477] Lustre: 3787:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages [37971.398203] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [38191.891205] LustreError: 35178:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651673021, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99eb63453a80/0x322afadd0ea7c0ba lrc: 3/1,0 mode: --/PR res: [0x200000408:0x1:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 35178 timeout: 0 lvb_type: 0 [38349.667201] Lustre: 3849:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb4792b180 x1731770935641280/t42961884363(0) o35->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:754/0 lens 392/456 e 15 to 0 dl 1651673484 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [38495.242067] Lustre: soaked-MDT0000: Client 7ad30715-59f4-4c17-88bd-2c39bf00253d (at 192.168.1.126@o2ib) reconnecting [38495.253871] Lustre: Skipped 7 previous similar messages [38552.292169] Lustre: 3511:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673675/real 0] req@ffff99ec40036c00 x1731866522217216/t0(0) o13->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673682 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38552.323860] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38553.268167] Lustre: 3513:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673676/real 0] req@ffff99eb4a68ec00 x1731866522217664/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673683 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38553.299848] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38555.396294] Lustre: 3516:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673678/real 0] req@ffff99eb6bd44380 x1731866522218048/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673685 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38555.427951] Lustre: 3516:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [38555.438697] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38555.457355] Lustre: Skipped 1 previous similar message [38564.775693] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [38564.787945] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (21): c: 0, oc: 0, rc: 8 [38572.449423] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [38572.461218] Lustre: Skipped 1 previous similar message [38753.127007] Lustre: MGS: haven't heard from client 303621b9-bdff-4c16-9341-be1b54272978 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e79980c800, cur 1651673883 expire 1651673733 last 1651673655 [38765.461979] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0000_UUID (at 192.168.1.104@o2ib) in 240 seconds. I think it's dead, and I am evicting it. exp ffff99e799808400, cur 1651673895 expire 1651673745 last 1651673655 [38816.786825] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 4 seconds [38816.798192] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 57 previous similar messages [38832.787551] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 2 seconds [38832.798917] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [38864.788965] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 29 seconds [38864.800437] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [38928.791790] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 93 seconds [38928.803261] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [39095.172111] Lustre: mdt01_022: service thread pid 35178 was inactive for 1203.240 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [39095.193195] Pid: 35178, comm: mdt01_022 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [39095.204302] Call Trace: [39095.207186] [<0>] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [39095.213380] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [39095.219878] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [39095.225954] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [39095.232224] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [39095.238382] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [39095.244166] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [39095.249562] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [39095.255282] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [39095.261285] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [39095.267688] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [39095.273003] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [39095.279207] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [39095.286251] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [39095.291712] [<0>] kthread+0xd1/0xe0 [39095.295618] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [39095.301404] [<0>] 0xfffffffffffffffe [39096.334372] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [39096.346160] Lustre: Skipped 4 previous similar messages [39164.561491] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39173.548182] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [39173.559968] Lustre: Skipped 3 previous similar messages [39263.278068] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39358.528400] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39404.342605] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39418.546991] LustreError: 11-0: soaked-OST0008-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39418.559603] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39424.995285] LustreError: 11-0: soaked-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39425.007865] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39427.320069] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39427.661464] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [39429.395456] LustreError: 11-0: soaked-OST000c-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39429.408070] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39588.537777] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [39588.549284] Lustre: Skipped 2 previous similar messages [39697.429122] Lustre: soaked-MDT0000: Client 0f628bfa-3bc1-48b7-a4a4-37175642f976 (at 192.168.1.135@o2ib) reconnecting [39697.440929] Lustre: Skipped 3 previous similar messages [39774.605233] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [39774.617016] Lustre: Skipped 3 previous similar messages [40298.524117] Lustre: soaked-MDT0000: Client d85fd948-cdf6-4a2e-8806-76ec090bacdc (at 192.168.1.136@o2ib) reconnecting [40298.535890] Lustre: Skipped 4 previous similar messages [40899.622432] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [40899.634229] Lustre: Skipped 8 previous similar messages [41273.316599] Lustre: mdt01_017: service thread pid 8314 was inactive for 200.574 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [41273.316841] Pid: 5287, comm: mdt00_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41273.316842] Call Trace: [41273.316959] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [41273.317018] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [41273.317049] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [41273.317072] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [41273.317095] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [41273.317117] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [41273.317138] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [41273.317160] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [41273.317214] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [41273.317275] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [41273.317357] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [41273.317435] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41273.317505] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41273.317572] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41273.317593] [<0>] kthread+0xd1/0xe0 [41273.317600] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41273.317631] [<0>] 0xfffffffffffffffe [41273.448674] Lustre: Skipped 1 previous similar message [41273.454424] Pid: 8314, comm: mdt01_017 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41273.465422] Call Trace: [41273.468254] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [41273.474418] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [41273.480892] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [41273.486939] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [41273.493196] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [41273.499354] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [41273.505122] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [41273.510502] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [41273.516210] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [41273.522188] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [41273.528580] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [41273.533880] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41273.540043] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41273.547079] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41273.552559] [<0>] kthread+0xd1/0xe0 [41273.556493] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41273.562278] [<0>] 0xfffffffffffffffe [41372.746038] LustreError: 8314:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651676202, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99ebf59469c0/0x322afadd0ea8d7d3 lrc: 3/1,0 mode: --/PR res: [0x200000408:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 8314 timeout: 0 lvb_type: 0 [41372.790593] LustreError: 8314:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) Skipped 1 previous similar message [41500.719617] Lustre: soaked-MDT0000: Client 2b5be8a8-9023-43d6-8098-538f23791711 (at 192.168.1.122@o2ib) reconnecting [41500.731410] Lustre: Skipped 8 previous similar messages [41597.939063] Lustre: mdt_rdpg01_006: service thread pid 3849 was inactive for 200.366 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [41597.960454] Pid: 3849, comm: mdt_rdpg01_006 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41597.971939] Call Trace: [41597.974732] [<0>] call_rwsem_down_write_failed+0x17/0x30 [41597.980748] [<0>] llog_cat_add_rec+0x11d/0x980 [obdclass] [41597.986838] [<0>] llog_add+0x182/0x1f0 [obdclass] [41597.992225] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [41597.998278] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [41598.003984] [<0>] lod_trans_stop+0x25c/0x340 [lod] [41598.009394] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [41598.014708] [<0>] mdd_attr_set+0x81c/0x1070 [mdd] [41598.020060] [<0>] mdt_mfd_close+0x771/0xbb0 [mdt] [41598.025330] [<0>] mdt_close_internal+0x141/0x240 [mdt] [41598.031102] [<0>] mdt_close+0x291/0x900 [mdt] [41598.036027] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41598.042224] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41598.049276] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41598.054740] [<0>] kthread+0xd1/0xe0 [41598.058655] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41598.064439] [<0>] 0xfffffffffffffffe [41667.830205] Lustre: 3770:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec3df5e300 x1731767650932736/t0(0) o101->db61db00-0999-4840-ab32-048bf6b05215@192.168.1.119@o2ib:297/0 lens 576/4112 e 24 to 0 dl 1651676802 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [41954.306953] Lustre: 3780:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/-282), not sending early reply req@ffff99e815a2d100 x1731764235741888/t0(0) o101->7ad30715-59f4-4c17-88bd-2c39bf00253d@192.168.1.126@o2ib:583/0 lens 576/4112 e 18 to 0 dl 1651677088 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [41992.420637] Lustre: 3568:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec41faad00 x1731770952040128/t42961884399(0) o35->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:622/0 lens 392/456 e 17 to 0 dl 1651677127 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [42101.813123] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [42101.824904] Lustre: Skipped 8 previous similar messages [42403.006938] Lustre: 3492:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651677525/real 0] req@ffff99e813d79200 x1731866524284544/t0(0) o41->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651677532 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [42403.038697] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [42410.166601] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42410.186431] LustreError: Skipped 269 previous similar messages [42413.946393] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [42413.958647] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (23): c: 6, oc: 0, rc: 8 [42442.240940] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.122@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42442.260777] LustreError: Skipped 146 previous similar messages [42506.476958] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.135@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42506.496789] LustreError: Skipped 464 previous similar messages [42576.382340] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42576.907687] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42576.918200] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42577.429515] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42577.440036] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42577.951763] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42577.962354] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42578.474456] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42578.484981] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42578.998129] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42579.008658] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42579.441223] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [42579.449919] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42579.868529] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [42580.395524] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.405858] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42580.413451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42580.420176] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42580.430094] blk_update_request: critical target error, dev sdh, sector 6986547072 [42580.438488] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42580.523943] sd 0:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.534259] sd 0:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current] [42580.541941] sd 0:0:0:0: [sdc] tag#1 <>ASC=0x94 ASCQ=0x1 [42580.548653] sd 0:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 b1 23 78 00 00 00 08 00 00 [42580.558660] blk_update_request: I/O error, dev sdc, sector 11608952 [42580.565750] device-mapper: multipath: Failing path 8:32. [42580.968604] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.978940] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42580.986532] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42580.993272] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42581.003186] blk_update_request: critical target error, dev sdh, sector 6986547072 [42581.011581] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42581.020062] Buffer I/O error on dev dm-1, logical block 873318384, async page read [42581.535221] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42581.545550] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42581.553160] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42581.559916] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [42581.569847] blk_update_request: critical target error, dev sdh, sector 0 [42581.577359] blk_update_request: critical target error, dev dm-1, sector 0 [42582.085450] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42582.095779] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42582.103373] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42582.110085] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42582.120003] blk_update_request: critical target error, dev sdh, sector 0 [42582.127525] blk_update_request: critical target error, dev dm-1, sector 0 [42582.135130] Buffer I/O error on dev dm-1, logical block 0, async page read [42582.651985] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42582.662310] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42582.669894] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42582.676612] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42582.686527] blk_update_request: critical target error, dev sdh, sector 6986547192 [42583.202021] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42583.212342] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42583.219935] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42583.226657] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42583.236908] Buffer I/O error on dev dm-1, logical block 873318399, async page read [42583.752073] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42583.762397] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42583.769994] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42583.776702] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42583.786643] Buffer I/O error on dev dm-1, logical block 0, async page read [42584.302096] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42584.312418] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42584.320012] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42584.326729] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42584.336684] Buffer I/O error on dev dm-1, logical block 0, async page read [42584.852140] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42584.862462] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42584.870055] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42584.876766] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [42584.886713] Buffer I/O error on dev dm-1, logical block 3, async page read [42585.418846] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42585.429170] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42585.436755] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42585.443473] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42585.453342] device-mapper: multipath: Reinstating path 8:32. [42585.459715] blk_update_request: 9 callbacks suppressed [42585.465457] blk_update_request: critical target error, dev sdh, sector 6986547072 [42585.473858] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42585.985526] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42585.995835] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42586.003451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42586.010178] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42586.020126] blk_update_request: critical target error, dev sdh, sector 6986547072 [42586.028566] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42586.037053] Buffer I/O error on dev dm-1, logical block 873318384, async page read [42586.552218] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42586.562532] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42586.570123] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42586.576835] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [42586.586751] blk_update_request: critical target error, dev sdh, sector 0 [42586.594275] blk_update_request: critical target error, dev dm-1, sector 0 [42587.118925] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42587.129242] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42587.136827] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42587.143546] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42587.153463] blk_update_request: critical target error, dev sdh, sector 0 [42587.160993] blk_update_request: critical target error, dev dm-1, sector 0 [42587.168604] Buffer I/O error on dev dm-1, logical block 0, async page read [42587.685619] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42587.695928] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42587.703538] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42587.710259] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42587.720180] blk_update_request: critical target error, dev sdh, sector 6986547192 [42587.728565] blk_update_request: critical target error, dev dm-1, sector 6986547192 [42588.252357] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42588.262668] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42588.270253] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42588.276963] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42588.286912] Buffer I/O error on dev dm-1, logical block 873318399, async page read [42588.802375] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42588.812684] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42588.820268] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42588.826980] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42588.836923] Buffer I/O error on dev dm-1, logical block 0, async page read [42589.352640] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42589.362948] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42589.370553] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42589.377279] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42589.387242] Buffer I/O error on dev dm-1, logical block 0, async page read [42589.902552] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42589.912871] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42589.920465] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42589.927198] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [42589.937150] Buffer I/O error on dev dm-1, logical block 3, async page read [42606.640932] Lustre: MGS: haven't heard from client b760fa03-2988-4278-b5a0-bfb777a9f6dc (at 192.168.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e82ee57400, cur 1651677736 expire 1651677586 last 1651677509 [42606.664657] Lustre: Skipped 3 previous similar messages [42634.828784] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.104@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42634.848625] LustreError: Skipped 931 previous similar messages [42702.883462] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [42702.895249] Lustre: Skipped 5 previous similar messages [42745.545235] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [42763.811166] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [43063.849963] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [43063.862232] Lustre: Skipped 1 previous similar message [43303.992197] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [43304.003988] Lustre: Skipped 11 previous similar messages [43905.099474] Lustre: soaked-MDT0000: Client a6431ad6-ad1a-403b-b2a6-2dbd7e58ec7b (at 192.168.1.120@o2ib) reconnecting [43905.111260] Lustre: Skipped 14 previous similar messages [44506.196195] Lustre: soaked-MDT0000: Client d85fd948-cdf6-4a2e-8806-76ec090bacdc (at 192.168.1.136@o2ib) reconnecting [44506.207977] Lustre: Skipped 10 previous similar messages [45107.289525] Lustre: soaked-MDT0000: Client 0f628bfa-3bc1-48b7-a4a4-37175642f976 (at 192.168.1.135@o2ib) reconnecting [45107.301324] Lustre: Skipped 9 previous similar messages [45149.065039] Lustre: 3512:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651680271/real 0] req@ffff99eb4a76a400 x1731866525474560/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651680278 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [45149.096720] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [45150.089092] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651680272/real 0] req@ffff99eb6461cc80 x1731866525474880/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651680279 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [45150.120749] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [45150.131478] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [45150.150154] Lustre: Skipped 1 previous similar message [45161.068590] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [45161.080835] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (23): c: 0, oc: 0, rc: 8 [45347.733093] Lustre: MGS: haven't heard from client d8239bef-30bf-4106-a92b-3c8abed3501d (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec40743800, cur 1651680477 expire 1651680327 last 1651680250 [45349.885118] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0004_UUID (at 192.168.1.104@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e81071a800, cur 1651680479 expire 1651680329 last 1651680250 [45417.079958] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 1 seconds [45417.091325] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 63 previous similar messages [45433.080658] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 6 seconds [45433.092023] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [45465.082086] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 16 seconds [45465.093557] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 12 previous similar messages [45529.084933] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [45529.096300] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 22 previous similar messages [45708.386663] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [45708.398454] Lustre: Skipped 8 previous similar messages [45728.086808] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [45753.186976] SysRq : Show State [45753.190444] task PC stack pid father [45753.197094] systemd S ffff99e5b63a0000 0 1 0 0x00000000 [45753.205022] Call Trace: [45753.207771] [] schedule+0x29/0x70 [45753.213336] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45753.221222] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [45753.228811] [] schedule_hrtimeout_range+0x13/0x20 [45753.235929] [] ep_poll+0x23e/0x360 [45753.241589] [] ? wake_up_state+0x20/0x20 [45753.247818] [] SyS_epoll_wait+0xed/0x120 [45753.254063] [] ? system_call_after_swapgs+0x96/0x13a [45753.261458] [] system_call_fastpath+0x25/0x2a [45753.268172] [] ? system_call_after_swapgs+0xa2/0x13a [45753.275568] kthreadd S ffff99e5b63a1080 0 2 0 0x00000000 [45753.283478] Call Trace: [45753.286216] [] schedule+0x29/0x70 [45753.291786] [] kthreadd+0x315/0x320 [45753.297570] [] ? kthread_create_on_cpu+0x60/0x60 [45753.304576] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.311879] [] ? kthread_create_on_cpu+0x60/0x60 [45753.318887] kworker/0:0H S ffff99e5b63a3180 0 4 2 0x00000000 [45753.326816] Call Trace: [45753.329550] [] schedule+0x29/0x70 [45753.335109] [] worker_thread+0x1d9/0x3c0 [45753.341338] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45753.348633] [] kthread+0xd1/0xe0 [45753.354086] [] ? insert_kthread_work+0x40/0x40 [45753.360901] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.368203] [] ? insert_kthread_work+0x40/0x40 [45753.375016] ksoftirqd/0 S ffff99e5b63a5280 0 6 2 0x00000000 [45753.382941] Call Trace: [45753.385690] [] schedule+0x29/0x70 [45753.391248] [] smpboot_thread_fn+0xe2/0x1a0 [45753.397766] [] ? lg_double_unlock+0x40/0x40 [45753.404293] [] kthread+0xd1/0xe0 [45753.409742] [] ? insert_kthread_work+0x40/0x40 [45753.416558] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.423874] [] ? insert_kthread_work+0x40/0x40 [45753.430689] migration/0 S ffff99e5b63a6300 0 7 2 0x00000000 [45753.438600] Call Trace: [45753.441345] [] schedule+0x29/0x70 [45753.446901] [] smpboot_thread_fn+0xe2/0x1a0 [45753.453420] [] ? lg_double_unlock+0x40/0x40 [45753.459950] [] kthread+0xd1/0xe0 [45753.465410] [] ? insert_kthread_work+0x40/0x40 [45753.472219] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.479513] [] ? insert_kthread_work+0x40/0x40 [45753.486332] rcu_bh S ffff99e5b5c88000 0 8 2 0x00000000 [45753.494258] Call Trace: [45753.497004] [] schedule+0x29/0x70 [45753.502571] [] rcu_gp_kthread+0xb2/0x720 [45753.508828] [] ? __schedule+0x320/0x680 [45753.514958] [] ? wake_up_atomic_t+0x30/0x30 [45753.521479] [] ? rcu_gp_fqs+0x90/0x90 [45753.527417] [] kthread+0xd1/0xe0 [45753.532886] [] ? insert_kthread_work+0x40/0x40 [45753.539702] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.546999] [] ? insert_kthread_work+0x40/0x40 [45753.553812] rcu_sched S ffff99e5b5c89080 0 9 2 0x00000000 [45753.561729] Call Trace: [45753.564478] [] schedule+0x29/0x70 [45753.570034] [] schedule_timeout+0x168/0x2d0 [45753.576571] [] ? requeue_timers+0x170/0x170 [45753.583089] [] ? prepare_to_wait+0x56/0x90 [45753.589509] [] rcu_gp_kthread+0x359/0x720 [45753.595840] [] ? wake_up_atomic_t+0x30/0x30 [45753.602364] [] ? rcu_gp_fqs+0x90/0x90 [45753.608306] [] kthread+0xd1/0xe0 [45753.613757] [] ? insert_kthread_work+0x40/0x40 [45753.620573] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.627892] [] ? insert_kthread_work+0x40/0x40 [45753.634702] lru-add-drain S ffff99e5b5c8a100 0 10 2 0x00000000 [45753.642610] Call Trace: [45753.645347] [] schedule+0x29/0x70 [45753.650905] [] rescuer_thread+0x31e/0x3c0 [45753.657228] [] ? __schedule+0x320/0x680 [45753.663362] [] ? process_one_work+0x440/0x440 [45753.670074] [] kthread+0xd1/0xe0 [45753.675531] [] ? insert_kthread_work+0x40/0x40 [45753.682346] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.689645] [] ? insert_kthread_work+0x40/0x40 [45753.696459] watchdog/0 S ffff99e5b5c8b180 0 11 2 0x00000000 [45753.704369] Call Trace: [45753.707115] [] schedule+0x29/0x70 [45753.712670] [] smpboot_thread_fn+0xe2/0x1a0 [45753.719190] [] ? lg_double_unlock+0x40/0x40 [45753.725724] [] kthread+0xd1/0xe0 [45753.731177] [] ? insert_kthread_work+0x40/0x40 [45753.737986] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.745280] [] ? insert_kthread_work+0x40/0x40 [45753.752096] watchdog/1 S ffff99e5b5e71080 0 12 2 0x00000000 [45753.760028] Call Trace: [45753.762778] [] schedule+0x29/0x70 [45753.768333] [] smpboot_thread_fn+0xe2/0x1a0 [45753.774856] [] ? lg_double_unlock+0x40/0x40 [45753.781394] [] kthread+0xd1/0xe0 [45753.786854] [] ? insert_kthread_work+0x40/0x40 [45753.793664] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.800964] [] ? insert_kthread_work+0x40/0x40 [45753.807797] migration/1 S ffff99e5b5e72100 0 13 2 0x00000000 [45753.815706] Call Trace: [45753.818445] [] schedule+0x29/0x70 [45753.824002] [] smpboot_thread_fn+0xe2/0x1a0 [45753.830521] [] ? lg_double_unlock+0x40/0x40 [45753.837042] [] kthread+0xd1/0xe0 [45753.842493] [] ? insert_kthread_work+0x40/0x40 [45753.849309] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.856604] [] ? insert_kthread_work+0x40/0x40 [45753.863411] ksoftirqd/1 S ffff99e5b5e73180 0 14 2 0x00000000 [45753.871318] Call Trace: [45753.874064] [] schedule+0x29/0x70 [45753.879615] [] smpboot_thread_fn+0xe2/0x1a0 [45753.886126] [] ? lg_double_unlock+0x40/0x40 [45753.892647] [] kthread+0xd1/0xe0 [45753.898112] [] ? insert_kthread_work+0x40/0x40 [45753.904920] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.912227] [] ? insert_kthread_work+0x40/0x40 [45753.919056] kworker/1:0H S ffff99e5b5e75280 0 16 2 0x00000000 [45753.926996] Call Trace: [45753.929737] [] schedule+0x29/0x70 [45753.935289] [] worker_thread+0x1d9/0x3c0 [45753.941512] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45753.948831] [] kthread+0xd1/0xe0 [45753.954285] [] ? insert_kthread_work+0x40/0x40 [45753.961098] [] ret_from_fork_nospec_begin+0x21/0x21 [45753.968397] [] ? insert_kthread_work+0x40/0x40 [45753.975207] watchdog/2 S ffff99e5b5ea1080 0 18 2 0x00000000 [45753.983112] Call Trace: [45753.985861] [] schedule+0x29/0x70 [45753.991427] [] smpboot_thread_fn+0xe2/0x1a0 [45753.997958] [] ? lg_double_unlock+0x40/0x40 [45754.004504] [] kthread+0xd1/0xe0 [45754.009990] [] ? insert_kthread_work+0x40/0x40 [45754.016816] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.024111] [] ? insert_kthread_work+0x40/0x40 [45754.030923] migration/2 S ffff99e5b5ea2100 0 19 2 0x00000000 [45754.038831] Call Trace: [45754.041569] [] schedule+0x29/0x70 [45754.047121] [] smpboot_thread_fn+0xe2/0x1a0 [45754.053643] [] ? lg_double_unlock+0x40/0x40 [45754.060153] [] kthread+0xd1/0xe0 [45754.065606] [] ? insert_kthread_work+0x40/0x40 [45754.072419] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.079716] [] ? insert_kthread_work+0x40/0x40 [45754.086516] ksoftirqd/2 S ffff99e5b5ea3180 0 20 2 0x00000000 [45754.094423] Call Trace: [45754.097164] [] schedule+0x29/0x70 [45754.102715] [] smpboot_thread_fn+0xe2/0x1a0 [45754.109228] [] ? lg_double_unlock+0x40/0x40 [45754.115740] [] kthread+0xd1/0xe0 [45754.121194] [] ? insert_kthread_work+0x40/0x40 [45754.127997] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.135294] [] ? insert_kthread_work+0x40/0x40 [45754.142108] kworker/2:0H S ffff99e5b5ea5280 0 22 2 0x00000000 [45754.150059] Call Trace: [45754.152798] [] schedule+0x29/0x70 [45754.158342] [] worker_thread+0x1d9/0x3c0 [45754.164579] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45754.171887] [] kthread+0xd1/0xe0 [45754.177343] [] ? insert_kthread_work+0x40/0x40 [45754.184155] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.191457] [] ? insert_kthread_work+0x40/0x40 [45754.198281] watchdog/3 S ffff99e5b5ea6300 0 23 2 0x00000000 [45754.206202] Call Trace: [45754.208941] [] schedule+0x29/0x70 [45754.214506] [] smpboot_thread_fn+0xe2/0x1a0 [45754.221026] [] ? lg_double_unlock+0x40/0x40 [45754.227561] [] kthread+0xd1/0xe0 [45754.233015] [] ? insert_kthread_work+0x40/0x40 [45754.239828] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.247126] [] ? insert_kthread_work+0x40/0x40 [45754.253936] migration/3 R running task 0 24 2 0x00000000 [45754.261871] Call Trace: [45754.264622] [] schedule+0x29/0x70 [45754.270179] [] smpboot_thread_fn+0xe2/0x1a0 [45754.276714] [] ? lg_double_unlock+0x40/0x40 [45754.283242] [] kthread+0xd1/0xe0 [45754.288704] [] ? insert_kthread_work+0x40/0x40 [45754.295521] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.302835] [] ? insert_kthread_work+0x40/0x40 [45754.309636] ksoftirqd/3 R running task 0 25 2 0x00000000 [45754.317544] Call Trace: [45754.320285] [] schedule+0x29/0x70 [45754.325835] [] smpboot_thread_fn+0xe2/0x1a0 [45754.332357] [] ? lg_double_unlock+0x40/0x40 [45754.338878] [] kthread+0xd1/0xe0 [45754.344329] [] ? insert_kthread_work+0x40/0x40 [45754.351139] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.358443] [] ? insert_kthread_work+0x40/0x40 [45754.365267] kworker/3:0H S ffff99e5b547b180 0 27 2 0x00000000 [45754.373203] Call Trace: [45754.375951] [] schedule+0x29/0x70 [45754.381517] [] worker_thread+0x1d9/0x3c0 [45754.387770] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45754.395087] [] kthread+0xd1/0xe0 [45754.400555] [] ? insert_kthread_work+0x40/0x40 [45754.407372] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.414681] [] ? insert_kthread_work+0x40/0x40 [45754.421503] watchdog/4 S ffff99e5b547c200 0 28 2 0x00000000 [45754.429449] Call Trace: [45754.432194] [] schedule+0x29/0x70 [45754.437752] [] smpboot_thread_fn+0xe2/0x1a0 [45754.444289] [] ? lg_double_unlock+0x40/0x40 [45754.450819] [] kthread+0xd1/0xe0 [45754.456286] [] ? insert_kthread_work+0x40/0x40 [45754.463120] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.470429] [] ? insert_kthread_work+0x40/0x40 [45754.477253] migration/4 S ffff99e5b547d280 0 29 2 0x00000000 [45754.485195] Call Trace: [45754.487944] [] schedule+0x29/0x70 [45754.493515] [] smpboot_thread_fn+0xe2/0x1a0 [45754.500049] [] ? lg_double_unlock+0x40/0x40 [45754.506585] [] kthread+0xd1/0xe0 [45754.512076] [] ? insert_kthread_work+0x40/0x40 [45754.518925] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.526244] [] ? insert_kthread_work+0x40/0x40 [45754.533055] ksoftirqd/4 S ffff99e5b547e300 0 30 2 0x00000000 [45754.541003] Call Trace: [45754.543756] [] schedule+0x29/0x70 [45754.549322] [] smpboot_thread_fn+0xe2/0x1a0 [45754.555850] [] ? lg_double_unlock+0x40/0x40 [45754.562376] [] kthread+0xd1/0xe0 [45754.567837] [] ? insert_kthread_work+0x40/0x40 [45754.574662] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.581964] [] ? insert_kthread_work+0x40/0x40 [45754.588783] kworker/4:0H S ffff99e5b54f1080 0 32 2 0x00000000 [45754.596755] Call Trace: [45754.599508] [] schedule+0x29/0x70 [45754.605078] [] worker_thread+0x1d9/0x3c0 [45754.611321] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45754.618619] [] kthread+0xd1/0xe0 [45754.624080] [] ? insert_kthread_work+0x40/0x40 [45754.630924] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.638235] [] ? insert_kthread_work+0x40/0x40 [45754.645066] watchdog/5 S ffff99e5b54f2100 0 33 2 0x00000000 [45754.652996] Call Trace: [45754.655748] [] schedule+0x29/0x70 [45754.661311] [] smpboot_thread_fn+0xe2/0x1a0 [45754.667848] [] ? lg_double_unlock+0x40/0x40 [45754.674374] [] kthread+0xd1/0xe0 [45754.679843] [] ? insert_kthread_work+0x40/0x40 [45754.686663] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.693960] [] ? insert_kthread_work+0x40/0x40 [45754.700782] migration/5 S ffff99e5b54f3180 0 34 2 0x00000000 [45754.708748] Call Trace: [45754.711508] [] schedule+0x29/0x70 [45754.717079] [] smpboot_thread_fn+0xe2/0x1a0 [45754.723599] [] ? lg_double_unlock+0x40/0x40 [45754.730133] [] kthread+0xd1/0xe0 [45754.735593] [] ? insert_kthread_work+0x40/0x40 [45754.742424] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.749730] [] ? insert_kthread_work+0x40/0x40 [45754.756545] ksoftirqd/5 S ffff99e5b54f4200 0 35 2 0x00000000 [45754.764475] Call Trace: [45754.767221] [] schedule+0x29/0x70 [45754.772794] [] smpboot_thread_fn+0xe2/0x1a0 [45754.779336] [] ? lg_double_unlock+0x40/0x40 [45754.785869] [] kthread+0xd1/0xe0 [45754.791336] [] ? insert_kthread_work+0x40/0x40 [45754.798166] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.805482] [] ? insert_kthread_work+0x40/0x40 [45754.812305] kworker/5:0H S ffff99e5b54f6300 0 37 2 0x00000000 [45754.820257] Call Trace: [45754.823024] [] schedule+0x29/0x70 [45754.828576] [] worker_thread+0x1d9/0x3c0 [45754.834827] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45754.842137] [] kthread+0xd1/0xe0 [45754.847606] [] ? insert_kthread_work+0x40/0x40 [45754.854433] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.861744] [] ? insert_kthread_work+0x40/0x40 [45754.868551] watchdog/6 S ffff99e5b5560000 0 38 2 0x00000000 [45754.876480] Call Trace: [45754.879226] [] schedule+0x29/0x70 [45754.884791] [] smpboot_thread_fn+0xe2/0x1a0 [45754.891326] [] ? lg_double_unlock+0x40/0x40 [45754.897859] [] kthread+0xd1/0xe0 [45754.903319] [] ? insert_kthread_work+0x40/0x40 [45754.910153] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.917463] [] ? insert_kthread_work+0x40/0x40 [45754.924278] migration/6 S ffff99e5b5561080 0 39 2 0x00000000 [45754.932198] Call Trace: [45754.934972] [] schedule+0x29/0x70 [45754.940526] [] smpboot_thread_fn+0xe2/0x1a0 [45754.947059] [] ? lg_double_unlock+0x40/0x40 [45754.953586] [] kthread+0xd1/0xe0 [45754.959046] [] ? insert_kthread_work+0x40/0x40 [45754.965871] [] ret_from_fork_nospec_begin+0x21/0x21 [45754.973181] [] ? insert_kthread_work+0x40/0x40 [45754.980009] ksoftirqd/6 S ffff99e86a4fea40 0 40 2 0x00000000 [45754.987964] Call Trace: [45754.990718] [] schedule+0x29/0x70 [45754.996275] [] smpboot_thread_fn+0xe2/0x1a0 [45755.002801] [] ? lg_double_unlock+0x40/0x40 [45755.009330] [] kthread+0xd1/0xe0 [45755.014799] [] ? insert_kthread_work+0x40/0x40 [45755.021626] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.028960] [] ? insert_kthread_work+0x40/0x40 [45755.035784] kworker/6:0H S ffff99e5b5564200 0 42 2 0x00000000 [45755.043757] Call Trace: [45755.046510] [] schedule+0x29/0x70 [45755.052070] [] worker_thread+0x1d9/0x3c0 [45755.058300] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45755.065604] [] kthread+0xd1/0xe0 [45755.071064] [] ? insert_kthread_work+0x40/0x40 [45755.077882] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.085192] [] ? insert_kthread_work+0x40/0x40 [45755.092037] watchdog/7 S ffff99e5b5565280 0 43 2 0x00000000 [45755.099992] Call Trace: [45755.102757] [] schedule+0x29/0x70 [45755.108315] [] smpboot_thread_fn+0xe2/0x1a0 [45755.114840] [] ? lg_double_unlock+0x40/0x40 [45755.121367] [] kthread+0xd1/0xe0 [45755.126833] [] ? insert_kthread_work+0x40/0x40 [45755.133658] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.140988] [] ? insert_kthread_work+0x40/0x40 [45755.147859] migration/7 S ffff99e5b5566300 0 44 2 0x00000000 [45755.155857] Call Trace: [45755.158602] [] schedule+0x29/0x70 [45755.164171] [] smpboot_thread_fn+0xe2/0x1a0 [45755.170706] [] ? lg_double_unlock+0x40/0x40 [45755.177241] [] kthread+0xd1/0xe0 [45755.182708] [] ? insert_kthread_work+0x40/0x40 [45755.189533] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.196844] [] ? insert_kthread_work+0x40/0x40 [45755.203668] ksoftirqd/7 S ffff99ec6b9ce400 0 45 2 0x00000000 [45755.211606] Call Trace: [45755.214359] [] schedule+0x29/0x70 [45755.219921] [] smpboot_thread_fn+0xe2/0x1a0 [45755.226447] [] ? lg_double_unlock+0x40/0x40 [45755.232986] [] kthread+0xd1/0xe0 [45755.238444] [] ? insert_kthread_work+0x40/0x40 [45755.245271] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.252580] [] ? insert_kthread_work+0x40/0x40 [45755.259396] kworker/7:0H S ffff99e5b55b2100 0 47 2 0x00000000 [45755.267343] Call Trace: [45755.270089] [] schedule+0x29/0x70 [45755.275653] [] worker_thread+0x1d9/0x3c0 [45755.281901] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45755.289206] [] kthread+0xd1/0xe0 [45755.294667] [] ? insert_kthread_work+0x40/0x40 [45755.301493] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.308804] [] ? insert_kthread_work+0x40/0x40 [45755.315633] watchdog/8 S ffff99e8f67f3180 0 48 2 0x00000000 [45755.323568] Call Trace: [45755.326321] [] schedule+0x29/0x70 [45755.331883] [] smpboot_thread_fn+0xe2/0x1a0 [45755.338408] [] ? lg_double_unlock+0x40/0x40 [45755.344973] [] kthread+0xd1/0xe0 [45755.350446] [] ? insert_kthread_work+0x40/0x40 [45755.357279] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.364590] [] ? insert_kthread_work+0x40/0x40 [45755.371413] migration/8 S ffff99e8f5978000 0 49 2 0x00000000 [45755.379357] Call Trace: [45755.382094] [] schedule+0x29/0x70 [45755.387658] [] smpboot_thread_fn+0xe2/0x1a0 [45755.394185] [] ? lg_double_unlock+0x40/0x40 [45755.400705] [] kthread+0xd1/0xe0 [45755.406165] [] ? insert_kthread_work+0x40/0x40 [45755.412996] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.420323] [] ? insert_kthread_work+0x40/0x40 [45755.427154] ksoftirqd/8 S ffff99e8f5979080 0 50 2 0x00000000 [45755.435089] Call Trace: [45755.437849] [] schedule+0x29/0x70 [45755.443401] [] smpboot_thread_fn+0xe2/0x1a0 [45755.449929] [] ? lg_double_unlock+0x40/0x40 [45755.456463] [] kthread+0xd1/0xe0 [45755.461938] [] ? insert_kthread_work+0x40/0x40 [45755.468784] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.476094] [] ? insert_kthread_work+0x40/0x40 [45755.482925] kworker/8:0H S ffff99e8f597b180 0 52 2 0x00000000 [45755.490898] Call Trace: [45755.493657] [] schedule+0x29/0x70 [45755.499222] [] worker_thread+0x1d9/0x3c0 [45755.505462] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45755.512804] [] kthread+0xd1/0xe0 [45755.518261] [] ? insert_kthread_work+0x40/0x40 [45755.525086] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.532400] [] ? insert_kthread_work+0x40/0x40 [45755.539239] watchdog/9 S ffff99e8f597d280 0 54 2 0x00000000 [45755.547188] Call Trace: [45755.549973] [] schedule+0x29/0x70 [45755.555539] [] smpboot_thread_fn+0xe2/0x1a0 [45755.562074] [] ? lg_double_unlock+0x40/0x40 [45755.568602] [] kthread+0xd1/0xe0 [45755.574078] [] ? insert_kthread_work+0x40/0x40 [45755.580904] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.588201] [] ? insert_kthread_work+0x40/0x40 [45755.595044] migration/9 S ffff99e8f597e300 0 55 2 0x00000000 [45755.603028] Call Trace: [45755.605788] [] schedule+0x29/0x70 [45755.611347] [] smpboot_thread_fn+0xe2/0x1a0 [45755.617881] [] ? lg_double_unlock+0x40/0x40 [45755.624416] [] kthread+0xd1/0xe0 [45755.629875] [] ? insert_kthread_work+0x40/0x40 [45755.636704] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.644010] [] ? insert_kthread_work+0x40/0x40 [45755.650840] ksoftirqd/9 S ffff99e8f59d0000 0 56 2 0x00000000 [45755.658787] Call Trace: [45755.661548] [] schedule+0x29/0x70 [45755.667119] [] smpboot_thread_fn+0xe2/0x1a0 [45755.673655] [] ? lg_double_unlock+0x40/0x40 [45755.680182] [] kthread+0xd1/0xe0 [45755.685659] [] ? insert_kthread_work+0x40/0x40 [45755.692482] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.699787] [] ? insert_kthread_work+0x40/0x40 [45755.706616] kworker/9:0H S ffff99e8f59d2100 0 58 2 0x00000000 [45755.714590] Call Trace: [45755.717341] [] schedule+0x29/0x70 [45755.722907] [] worker_thread+0x1d9/0x3c0 [45755.729159] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45755.736477] [] kthread+0xd1/0xe0 [45755.741942] [] ? insert_kthread_work+0x40/0x40 [45755.748766] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.756077] [] ? insert_kthread_work+0x40/0x40 [45755.762906] watchdog/10 S ffff99e8f59d3180 0 59 2 0x00000000 [45755.770864] Call Trace: [45755.773617] [] schedule+0x29/0x70 [45755.779173] [] smpboot_thread_fn+0xe2/0x1a0 [45755.785701] [] ? lg_double_unlock+0x40/0x40 [45755.792239] [] kthread+0xd1/0xe0 [45755.797723] [] ? insert_kthread_work+0x40/0x40 [45755.804565] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.811895] [] ? insert_kthread_work+0x40/0x40 [45755.818727] migration/10 S ffff99e8f59d4200 0 60 2 0x00000000 [45755.826693] Call Trace: [45755.829443] [] schedule+0x29/0x70 [45755.835008] [] smpboot_thread_fn+0xe2/0x1a0 [45755.841534] [] ? lg_double_unlock+0x40/0x40 [45755.848070] [] kthread+0xd1/0xe0 [45755.853539] [] ? insert_kthread_work+0x40/0x40 [45755.860372] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.867690] [] ? insert_kthread_work+0x40/0x40 [45755.874527] ksoftirqd/10 S ffff99e8f59d5280 0 61 2 0x00000000 [45755.882471] Call Trace: [45755.885232] [] schedule+0x29/0x70 [45755.890797] [] smpboot_thread_fn+0xe2/0x1a0 [45755.897325] [] ? lg_double_unlock+0x40/0x40 [45755.903859] [] kthread+0xd1/0xe0 [45755.909327] [] ? insert_kthread_work+0x40/0x40 [45755.916156] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.923455] [] ? insert_kthread_work+0x40/0x40 [45755.930294] kworker/10:0H S ffff99e8f5a18000 0 63 2 0x00000000 [45755.938234] Call Trace: [45755.941004] [] schedule+0x29/0x70 [45755.946570] [] worker_thread+0x1d9/0x3c0 [45755.952808] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45755.960110] [] kthread+0xd1/0xe0 [45755.965581] [] ? insert_kthread_work+0x40/0x40 [45755.972407] [] ret_from_fork_nospec_begin+0x21/0x21 [45755.979719] [] ? insert_kthread_work+0x40/0x40 [45755.986543] watchdog/11 S ffff99e8f5a19080 0 64 2 0x00000000 [45755.994509] Call Trace: [45755.997254] [] schedule+0x29/0x70 [45756.002819] [] smpboot_thread_fn+0xe2/0x1a0 [45756.009353] [] ? lg_double_unlock+0x40/0x40 [45756.015887] [] kthread+0xd1/0xe0 [45756.021361] [] ? insert_kthread_work+0x40/0x40 [45756.028171] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.035483] [] ? insert_kthread_work+0x40/0x40 [45756.042304] migration/11 S ffff99e8f5a1a100 0 65 2 0x00000000 [45756.050242] Call Trace: [45756.053008] [] schedule+0x29/0x70 [45756.058582] [] smpboot_thread_fn+0xe2/0x1a0 [45756.065118] [] ? lg_double_unlock+0x40/0x40 [45756.071651] [] kthread+0xd1/0xe0 [45756.077110] [] ? insert_kthread_work+0x40/0x40 [45756.083936] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.091254] [] ? insert_kthread_work+0x40/0x40 [45756.098087] ksoftirqd/11 S ffff99e8f5a1b180 0 66 2 0x00000000 [45756.106045] Call Trace: [45756.108809] [] schedule+0x29/0x70 [45756.114376] [] smpboot_thread_fn+0xe2/0x1a0 [45756.120916] [] ? lg_double_unlock+0x40/0x40 [45756.127450] [] kthread+0xd1/0xe0 [45756.132918] [] ? insert_kthread_work+0x40/0x40 [45756.139744] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.147074] [] ? insert_kthread_work+0x40/0x40 [45756.153923] kworker/11:0H S ffff99e8f5a1d280 0 68 2 0x00000000 [45756.161908] Call Trace: [45756.164675] [] schedule+0x29/0x70 [45756.170236] [] worker_thread+0x1d9/0x3c0 [45756.176495] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45756.183816] [] kthread+0xd1/0xe0 [45756.189293] [] ? insert_kthread_work+0x40/0x40 [45756.196120] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.203417] [] ? insert_kthread_work+0x40/0x40 [45756.210231] watchdog/12 S ffff99e8f5a1e300 0 69 2 0x00000000 [45756.218141] Call Trace: [45756.220886] [] schedule+0x29/0x70 [45756.226444] [] smpboot_thread_fn+0xe2/0x1a0 [45756.233010] [] ? lg_double_unlock+0x40/0x40 [45756.239543] [] kthread+0xd1/0xe0 [45756.245041] [] ? insert_kthread_work+0x40/0x40 [45756.251874] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.259186] [] ? insert_kthread_work+0x40/0x40 [45756.266032] migration/12 S ffff99e8f5aa8000 0 70 2 0x00000000 [45756.273970] Call Trace: [45756.276728] [] schedule+0x29/0x70 [45756.282284] [] smpboot_thread_fn+0xe2/0x1a0 [45756.288813] [] ? lg_double_unlock+0x40/0x40 [45756.295349] [] kthread+0xd1/0xe0 [45756.300823] [] ? insert_kthread_work+0x40/0x40 [45756.307641] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.314971] [] ? insert_kthread_work+0x40/0x40 [45756.321801] ksoftirqd/12 S ffff99e8f5aa9080 0 71 2 0x00000000 [45756.329740] Call Trace: [45756.332499] [] schedule+0x29/0x70 [45756.338069] [] smpboot_thread_fn+0xe2/0x1a0 [45756.344603] [] ? lg_double_unlock+0x40/0x40 [45756.351138] [] kthread+0xd1/0xe0 [45756.356615] [] ? insert_kthread_work+0x40/0x40 [45756.363426] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.370736] [] ? insert_kthread_work+0x40/0x40 [45756.377567] kworker/12:0H S ffff99e8f5aab180 0 73 2 0x00000000 [45756.385505] Call Trace: [45756.388257] [] schedule+0x29/0x70 [45756.393823] [] worker_thread+0x1d9/0x3c0 [45756.400079] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45756.407389] [] kthread+0xd1/0xe0 [45756.412860] [] ? insert_kthread_work+0x40/0x40 [45756.419686] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.427019] [] ? insert_kthread_work+0x40/0x40 [45756.433842] watchdog/13 S ffff99e8f5aac200 0 74 2 0x00000000 [45756.441807] Call Trace: [45756.444577] [] schedule+0x29/0x70 [45756.450143] [] smpboot_thread_fn+0xe2/0x1a0 [45756.456678] [] ? lg_double_unlock+0x40/0x40 [45756.463197] [] kthread+0xd1/0xe0 [45756.468672] [] ? insert_kthread_work+0x40/0x40 [45756.475497] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.482817] [] ? insert_kthread_work+0x40/0x40 [45756.489639] migration/13 S ffff99e8f5aad280 0 75 2 0x00000000 [45756.497598] Call Trace: [45756.500357] [] schedule+0x29/0x70 [45756.505916] [] smpboot_thread_fn+0xe2/0x1a0 [45756.512444] [] ? lg_double_unlock+0x40/0x40 [45756.518986] [] kthread+0xd1/0xe0 [45756.524484] [] ? insert_kthread_work+0x40/0x40 [45756.531312] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.538623] [] ? insert_kthread_work+0x40/0x40 [45756.545446] ksoftirqd/13 S ffff99e8f5aae300 0 76 2 0x00000000 [45756.553377] Call Trace: [45756.556138] [] schedule+0x29/0x70 [45756.561703] [] smpboot_thread_fn+0xe2/0x1a0 [45756.568224] [] ? lg_double_unlock+0x40/0x40 [45756.574765] [] kthread+0xd1/0xe0 [45756.580240] [] ? insert_kthread_work+0x40/0x40 [45756.587087] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.594396] [] ? insert_kthread_work+0x40/0x40 [45756.601233] kworker/13:0H S ffff99e8f5ae9080 0 78 2 0x00000000 [45756.609187] Call Trace: [45756.611946] [] schedule+0x29/0x70 [45756.617504] [] worker_thread+0x1d9/0x3c0 [45756.623757] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45756.631091] [] kthread+0xd1/0xe0 [45756.636566] [] ? insert_kthread_work+0x40/0x40 [45756.643387] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.650698] [] ? insert_kthread_work+0x40/0x40 [45756.657529] watchdog/14 S ffff99e8f5aea100 0 79 2 0x00000000 [45756.665471] Call Trace: [45756.668208] [] schedule+0x29/0x70 [45756.673765] [] smpboot_thread_fn+0xe2/0x1a0 [45756.680298] [] ? lg_double_unlock+0x40/0x40 [45756.686822] [] kthread+0xd1/0xe0 [45756.692296] [] ? insert_kthread_work+0x40/0x40 [45756.699116] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.706428] [] ? insert_kthread_work+0x40/0x40 [45756.713257] migration/14 S ffff99e8f5aeb180 0 80 2 0x00000000 [45756.721168] Call Trace: [45756.723922] [] schedule+0x29/0x70 [45756.729480] [] smpboot_thread_fn+0xe2/0x1a0 [45756.736027] [] ? lg_double_unlock+0x40/0x40 [45756.742560] [] kthread+0xd1/0xe0 [45756.748050] [] ? insert_kthread_work+0x40/0x40 [45756.754874] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.762178] [] ? insert_kthread_work+0x40/0x40 [45756.769031] ksoftirqd/14 S ffff99e8f5aec200 0 81 2 0x00000000 [45756.776969] Call Trace: [45756.779729] [] schedule+0x29/0x70 [45756.785292] [] smpboot_thread_fn+0xe2/0x1a0 [45756.791817] [] ? lg_double_unlock+0x40/0x40 [45756.798339] [] kthread+0xd1/0xe0 [45756.803814] [] ? insert_kthread_work+0x40/0x40 [45756.810629] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.817934] [] ? insert_kthread_work+0x40/0x40 [45756.824778] kworker/14:0H S ffff99e8f5aee300 0 83 2 0x00000000 [45756.832745] Call Trace: [45756.835499] [] schedule+0x29/0x70 [45756.841056] [] worker_thread+0x1d9/0x3c0 [45756.847303] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45756.854616] [] kthread+0xd1/0xe0 [45756.860084] [] ? insert_kthread_work+0x40/0x40 [45756.866912] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.874218] [] ? insert_kthread_work+0x40/0x40 [45756.881059] watchdog/15 S ffff99e8f5b50000 0 84 2 0x00000000 [45756.889046] Call Trace: [45756.891806] [] schedule+0x29/0x70 [45756.897374] [] smpboot_thread_fn+0xe2/0x1a0 [45756.903911] [] ? lg_double_unlock+0x40/0x40 [45756.910447] [] kthread+0xd1/0xe0 [45756.915916] [] ? insert_kthread_work+0x40/0x40 [45756.922739] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.930049] [] ? insert_kthread_work+0x40/0x40 [45756.936878] migration/15 S ffff99ec6b9ce400 0 85 2 0x00000000 [45756.944837] Call Trace: [45756.947596] [] schedule+0x29/0x70 [45756.953155] [] smpboot_thread_fn+0xe2/0x1a0 [45756.959688] [] ? lg_double_unlock+0x40/0x40 [45756.966225] [] kthread+0xd1/0xe0 [45756.971694] [] ? insert_kthread_work+0x40/0x40 [45756.978516] [] ret_from_fork_nospec_begin+0x21/0x21 [45756.985821] [] ? insert_kthread_work+0x40/0x40 [45756.992645] ksoftirqd/15 S ffff99e8f5b52100 0 86 2 0x00000000 [45757.000588] Call Trace: [45757.003319] [] schedule+0x29/0x70 [45757.008889] [] smpboot_thread_fn+0xe2/0x1a0 [45757.015431] [] ? lg_double_unlock+0x40/0x40 [45757.021958] [] kthread+0xd1/0xe0 [45757.027432] [] ? insert_kthread_work+0x40/0x40 [45757.034249] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.041551] [] ? insert_kthread_work+0x40/0x40 [45757.048367] kworker/15:0H S ffff99e8f5b54200 0 88 2 0x00000000 [45757.056327] Call Trace: [45757.059077] [] schedule+0x29/0x70 [45757.064658] [] worker_thread+0x1d9/0x3c0 [45757.070902] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45757.078198] [] kthread+0xd1/0xe0 [45757.083659] [] ? insert_kthread_work+0x40/0x40 [45757.090478] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.097791] [] ? insert_kthread_work+0x40/0x40 [45757.104614] watchdog/16 S ffff99e5b55b6300 0 89 2 0x00000000 [45757.112556] Call Trace: [45757.115303] [] schedule+0x29/0x70 [45757.120867] [] smpboot_thread_fn+0xe2/0x1a0 [45757.127402] [] ? lg_double_unlock+0x40/0x40 [45757.133936] [] kthread+0xd1/0xe0 [45757.139411] [] ? insert_kthread_work+0x40/0x40 [45757.146228] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.153537] [] ? insert_kthread_work+0x40/0x40 [45757.160360] migration/16 S ffff99e5b55b5280 0 90 2 0x00000000 [45757.168288] Call Trace: [45757.171047] [] schedule+0x29/0x70 [45757.176609] [] smpboot_thread_fn+0xe2/0x1a0 [45757.183128] [] ? lg_double_unlock+0x40/0x40 [45757.189660] [] kthread+0xd1/0xe0 [45757.195123] [] ? insert_kthread_work+0x40/0x40 [45757.201940] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.209244] [] ? insert_kthread_work+0x40/0x40 [45757.216061] ksoftirqd/16 S ffff99e5b55b4200 0 91 2 0x00000000 [45757.223997] Call Trace: [45757.226744] [] schedule+0x29/0x70 [45757.232300] [] smpboot_thread_fn+0xe2/0x1a0 [45757.238832] [] ? lg_double_unlock+0x40/0x40 [45757.245363] [] kthread+0xd1/0xe0 [45757.250832] [] ? insert_kthread_work+0x40/0x40 [45757.257650] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.264967] [] ? insert_kthread_work+0x40/0x40 [45757.271798] kworker/16:0 S ffff99e5b55b3180 0 92 2 0x00000000 [45757.279751] Call Trace: [45757.282498] [] schedule+0x29/0x70 [45757.288058] [] worker_thread+0x1d9/0x3c0 [45757.294294] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45757.301602] [] kthread+0xd1/0xe0 [45757.307072] [] ? insert_kthread_work+0x40/0x40 [45757.313888] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.321199] [] ? insert_kthread_work+0x40/0x40 [45757.328017] kworker/16:0H S ffff99e5b5640000 0 93 2 0x00000000 [45757.335969] Call Trace: [45757.338727] [] schedule+0x29/0x70 [45757.344284] [] worker_thread+0x1d9/0x3c0 [45757.350527] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45757.357839] [] kthread+0xd1/0xe0 [45757.363293] [] ? insert_kthread_work+0x40/0x40 [45757.370123] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.377427] [] ? insert_kthread_work+0x40/0x40 [45757.384236] watchdog/17 S ffff99e5b5641080 0 94 2 0x00000000 [45757.392158] Call Trace: [45757.394918] [] schedule+0x29/0x70 [45757.400474] [] smpboot_thread_fn+0xe2/0x1a0 [45757.407018] [] ? lg_double_unlock+0x40/0x40 [45757.413545] [] kthread+0xd1/0xe0 [45757.419015] [] ? insert_kthread_work+0x40/0x40 [45757.425834] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.433129] [] ? insert_kthread_work+0x40/0x40 [45757.439954] migration/17 S ffff99e5b5642100 0 95 2 0x00000000 [45757.447888] Call Trace: [45757.450633] [] schedule+0x29/0x70 [45757.456188] [] smpboot_thread_fn+0xe2/0x1a0 [45757.462730] [] ? lg_double_unlock+0x40/0x40 [45757.469271] [] kthread+0xd1/0xe0 [45757.474739] [] ? insert_kthread_work+0x40/0x40 [45757.481564] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.488876] [] ? insert_kthread_work+0x40/0x40 [45757.495700] ksoftirqd/17 S ffff99e86a4fddc0 0 96 2 0x00000000 [45757.503634] Call Trace: [45757.506371] [] schedule+0x29/0x70 [45757.511936] [] smpboot_thread_fn+0xe2/0x1a0 [45757.518488] [] ? lg_double_unlock+0x40/0x40 [45757.525031] [] kthread+0xd1/0xe0 [45757.530513] [] ? insert_kthread_work+0x40/0x40 [45757.537337] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.544646] [] ? insert_kthread_work+0x40/0x40 [45757.551463] kworker/17:0H S ffff99e5b5645280 0 98 2 0x00000000 [45757.559403] Call Trace: [45757.562152] [] schedule+0x29/0x70 [45757.567709] [] worker_thread+0x1d9/0x3c0 [45757.573952] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45757.581263] [] kthread+0xd1/0xe0 [45757.586739] [] ? insert_kthread_work+0x40/0x40 [45757.593566] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.600877] [] ? insert_kthread_work+0x40/0x40 [45757.607695] watchdog/18 S ffff99e5b5646300 0 99 2 0x00000000 [45757.615631] Call Trace: [45757.618377] [] schedule+0x29/0x70 [45757.623942] [] smpboot_thread_fn+0xe2/0x1a0 [45757.630475] [] ? lg_double_unlock+0x40/0x40 [45757.637012] [] kthread+0xd1/0xe0 [45757.642473] [] ? insert_kthread_work+0x40/0x40 [45757.649291] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.656612] [] ? insert_kthread_work+0x40/0x40 [45757.663449] migration/18 S ffff99e5b56c8000 0 100 2 0x00000000 [45757.671393] Call Trace: [45757.674146] [] schedule+0x29/0x70 [45757.679709] [] smpboot_thread_fn+0xe2/0x1a0 [45757.686243] [] ? lg_double_unlock+0x40/0x40 [45757.692777] [] kthread+0xd1/0xe0 [45757.698236] [] ? insert_kthread_work+0x40/0x40 [45757.705079] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.712397] [] ? insert_kthread_work+0x40/0x40 [45757.719227] ksoftirqd/18 S ffff99e86d370640 0 101 2 0x00000000 [45757.727177] Call Trace: [45757.729913] [] schedule+0x29/0x70 [45757.735470] [] smpboot_thread_fn+0xe2/0x1a0 [45757.741997] [] ? lg_double_unlock+0x40/0x40 [45757.748530] [] kthread+0xd1/0xe0 [45757.753991] [] ? insert_kthread_work+0x40/0x40 [45757.760816] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.768131] [] ? insert_kthread_work+0x40/0x40 [45757.774954] kworker/18:0H S ffff99e5b56cb180 0 103 2 0x00000000 [45757.782889] Call Trace: [45757.785643] [] schedule+0x29/0x70 [45757.791226] [] worker_thread+0x1d9/0x3c0 [45757.797471] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45757.804780] [] kthread+0xd1/0xe0 [45757.810234] [] ? insert_kthread_work+0x40/0x40 [45757.817072] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.824374] [] ? insert_kthread_work+0x40/0x40 [45757.831198] watchdog/19 S ffff99e5b56cc200 0 104 2 0x00000000 [45757.839146] Call Trace: [45757.841900] [] schedule+0x29/0x70 [45757.847461] [] smpboot_thread_fn+0xe2/0x1a0 [45757.854017] [] ? lg_double_unlock+0x40/0x40 [45757.860543] [] kthread+0xd1/0xe0 [45757.866015] [] ? insert_kthread_work+0x40/0x40 [45757.872834] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.880146] [] ? insert_kthread_work+0x40/0x40 [45757.886968] migration/19 S ffff99e5b56cd280 0 105 2 0x00000000 [45757.894891] Call Trace: [45757.897638] [] schedule+0x29/0x70 [45757.903194] [] smpboot_thread_fn+0xe2/0x1a0 [45757.909735] [] ? lg_double_unlock+0x40/0x40 [45757.916275] [] kthread+0xd1/0xe0 [45757.921734] [] ? insert_kthread_work+0x40/0x40 [45757.928560] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.935870] [] ? insert_kthread_work+0x40/0x40 [45757.942693] ksoftirqd/19 S ffff99e86c47e400 0 106 2 0x00000000 [45757.950618] Call Trace: [45757.953364] [] schedule+0x29/0x70 [45757.958922] [] smpboot_thread_fn+0xe2/0x1a0 [45757.965457] [] ? lg_double_unlock+0x40/0x40 [45757.972000] [] kthread+0xd1/0xe0 [45757.977475] [] ? insert_kthread_work+0x40/0x40 [45757.984301] [] ret_from_fork_nospec_begin+0x21/0x21 [45757.991611] [] ? insert_kthread_work+0x40/0x40 [45757.998434] kworker/19:0H S ffff99e5b5721080 0 108 2 0x00000000 [45758.006372] Call Trace: [45758.009118] [] schedule+0x29/0x70 [45758.014696] [] worker_thread+0x1d9/0x3c0 [45758.020940] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45758.028241] [] kthread+0xd1/0xe0 [45758.033701] [] ? insert_kthread_work+0x40/0x40 [45758.040534] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.047845] [] ? insert_kthread_work+0x40/0x40 [45758.054661] watchdog/20 S ffff99e5b5722100 0 109 2 0x00000000 [45758.062595] Call Trace: [45758.065341] [] schedule+0x29/0x70 [45758.070899] [] smpboot_thread_fn+0xe2/0x1a0 [45758.077419] [] ? lg_double_unlock+0x40/0x40 [45758.083945] [] kthread+0xd1/0xe0 [45758.089411] [] ? insert_kthread_work+0x40/0x40 [45758.096230] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.103534] [] ? insert_kthread_work+0x40/0x40 [45758.110364] migration/20 S ffff99e5b5723180 0 110 2 0x00000000 [45758.118284] Call Trace: [45758.121036] [] schedule+0x29/0x70 [45758.126601] [] smpboot_thread_fn+0xe2/0x1a0 [45758.133121] [] ? lg_double_unlock+0x40/0x40 [45758.139656] [] kthread+0xd1/0xe0 [45758.145146] [] ? insert_kthread_work+0x40/0x40 [45758.151971] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.159280] [] ? insert_kthread_work+0x40/0x40 [45758.166117] ksoftirqd/20 S ffff99e5b5724200 0 111 2 0x00000000 [45758.174094] Call Trace: [45758.176839] [] schedule+0x29/0x70 [45758.182397] [] smpboot_thread_fn+0xe2/0x1a0 [45758.188925] [] ? lg_double_unlock+0x40/0x40 [45758.195458] [] kthread+0xd1/0xe0 [45758.200926] [] ? insert_kthread_work+0x40/0x40 [45758.207751] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.215064] [] ? insert_kthread_work+0x40/0x40 [45758.221913] kworker/20:0H S ffff99e5b5726300 0 113 2 0x00000000 [45758.229875] Call Trace: [45758.232636] [] schedule+0x29/0x70 [45758.238186] [] worker_thread+0x1d9/0x3c0 [45758.244430] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45758.251732] [] kthread+0xd1/0xe0 [45758.257200] [] ? insert_kthread_work+0x40/0x40 [45758.264026] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.271332] [] ? insert_kthread_work+0x40/0x40 [45758.278164] watchdog/21 S ffff99e5b5788000 0 114 2 0x00000000 [45758.286106] Call Trace: [45758.288851] [] schedule+0x29/0x70 [45758.294401] [] smpboot_thread_fn+0xe2/0x1a0 [45758.300943] [] ? lg_double_unlock+0x40/0x40 [45758.307473] [] kthread+0xd1/0xe0 [45758.312934] [] ? insert_kthread_work+0x40/0x40 [45758.319751] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.327057] [] ? insert_kthread_work+0x40/0x40 [45758.333878] migration/21 S ffff99e5b5789080 0 115 2 0x00000000 [45758.341818] Call Trace: [45758.344572] [] schedule+0x29/0x70 [45758.350150] [] smpboot_thread_fn+0xe2/0x1a0 [45758.356685] [] ? lg_double_unlock+0x40/0x40 [45758.363211] [] kthread+0xd1/0xe0 [45758.368680] [] ? insert_kthread_work+0x40/0x40 [45758.375498] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.382799] [] ? insert_kthread_work+0x40/0x40 [45758.389620] ksoftirqd/21 S ffff99e86a4f8000 0 116 2 0x00000000 [45758.397550] Call Trace: [45758.400297] [] schedule+0x29/0x70 [45758.405861] [] smpboot_thread_fn+0xe2/0x1a0 [45758.412396] [] ? lg_double_unlock+0x40/0x40 [45758.418936] [] kthread+0xd1/0xe0 [45758.424410] [] ? insert_kthread_work+0x40/0x40 [45758.431242] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.438544] [] ? insert_kthread_work+0x40/0x40 [45758.445358] kworker/21:0H S ffff99e5b578c200 0 118 2 0x00000000 [45758.453287] Call Trace: [45758.456043] [] schedule+0x29/0x70 [45758.461609] [] worker_thread+0x1d9/0x3c0 [45758.467840] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45758.475163] [] kthread+0xd1/0xe0 [45758.480613] [] ? insert_kthread_work+0x40/0x40 [45758.487444] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.494772] [] ? insert_kthread_work+0x40/0x40 [45758.501594] watchdog/22 S ffff99e5b578d280 0 119 2 0x00000000 [45758.509520] Call Trace: [45758.512266] [] schedule+0x29/0x70 [45758.517818] [] smpboot_thread_fn+0xe2/0x1a0 [45758.524353] [] ? lg_double_unlock+0x40/0x40 [45758.530889] [] kthread+0xd1/0xe0 [45758.536357] [] ? insert_kthread_work+0x40/0x40 [45758.543178] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.550496] [] ? insert_kthread_work+0x40/0x40 [45758.557330] migration/22 S ffff99e5b578e300 0 120 2 0x00000000 [45758.565277] Call Trace: [45758.568016] [] schedule+0x29/0x70 [45758.573579] [] smpboot_thread_fn+0xe2/0x1a0 [45758.580131] [] ? lg_double_unlock+0x40/0x40 [45758.586657] [] kthread+0xd1/0xe0 [45758.592132] [] ? insert_kthread_work+0x40/0x40 [45758.598958] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.606269] [] ? insert_kthread_work+0x40/0x40 [45758.613079] ksoftirqd/22 S ffff99e5b57f0000 0 121 2 0x00000000 [45758.621031] Call Trace: [45758.623778] [] schedule+0x29/0x70 [45758.629341] [] smpboot_thread_fn+0xe2/0x1a0 [45758.635877] [] ? lg_double_unlock+0x40/0x40 [45758.642398] [] kthread+0xd1/0xe0 [45758.647859] [] ? insert_kthread_work+0x40/0x40 [45758.654681] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.661999] [] ? insert_kthread_work+0x40/0x40 [45758.668822] kworker/22:0H S ffff99e5b57f2100 0 123 2 0x00000000 [45758.676771] Call Trace: [45758.679524] [] schedule+0x29/0x70 [45758.685080] [] worker_thread+0x1d9/0x3c0 [45758.691330] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45758.698639] [] kthread+0xd1/0xe0 [45758.704118] [] ? insert_kthread_work+0x40/0x40 [45758.710949] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.718243] [] ? insert_kthread_work+0x40/0x40 [45758.725059] watchdog/23 S ffff99e5b57f3180 0 124 2 0x00000000 [45758.732969] Call Trace: [45758.735722] [] schedule+0x29/0x70 [45758.741269] [] smpboot_thread_fn+0xe2/0x1a0 [45758.747785] [] ? lg_double_unlock+0x40/0x40 [45758.754326] [] kthread+0xd1/0xe0 [45758.759784] [] ? insert_kthread_work+0x40/0x40 [45758.766603] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.773914] [] ? insert_kthread_work+0x40/0x40 [45758.780727] migration/23 S ffff99e5b57f4200 0 125 2 0x00000000 [45758.788655] Call Trace: [45758.791404] [] schedule+0x29/0x70 [45758.796962] [] smpboot_thread_fn+0xe2/0x1a0 [45758.803500] [] ? lg_double_unlock+0x40/0x40 [45758.810114] [] kthread+0xd1/0xe0 [45758.815583] [] ? insert_kthread_work+0x40/0x40 [45758.822394] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.829704] [] ? insert_kthread_work+0x40/0x40 [45758.836521] ksoftirqd/23 S ffff99e5b57f5280 0 126 2 0x00000000 [45758.844449] Call Trace: [45758.847197] [] schedule+0x29/0x70 [45758.852747] [] smpboot_thread_fn+0xe2/0x1a0 [45758.859285] [] ? lg_double_unlock+0x40/0x40 [45758.865820] [] kthread+0xd1/0xe0 [45758.871280] [] ? insert_kthread_work+0x40/0x40 [45758.878094] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.885407] [] ? insert_kthread_work+0x40/0x40 [45758.892225] kworker/23:0H S ffff99e5b5048000 0 128 2 0x00000000 [45758.900154] Call Trace: [45758.902892] [] schedule+0x29/0x70 [45758.908450] [] worker_thread+0x1d9/0x3c0 [45758.914685] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45758.921990] [] kthread+0xd1/0xe0 [45758.927442] [] ? insert_kthread_work+0x40/0x40 [45758.934260] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.941576] [] ? insert_kthread_work+0x40/0x40 [45758.948386] watchdog/24 S ffff99e8f5b56300 0 129 2 0x00000000 [45758.956298] Call Trace: [45758.959052] [] schedule+0x29/0x70 [45758.964617] [] smpboot_thread_fn+0xe2/0x1a0 [45758.971151] [] ? lg_double_unlock+0x40/0x40 [45758.977678] [] kthread+0xd1/0xe0 [45758.983140] [] ? insert_kthread_work+0x40/0x40 [45758.989968] [] ret_from_fork_nospec_begin+0x21/0x21 [45758.997280] [] ? insert_kthread_work+0x40/0x40 [45759.004140] migration/24 S ffff99e8f5b55280 0 130 2 0x00000000 [45759.012080] Call Trace: [45759.014834] [] schedule+0x29/0x70 [45759.020392] [] smpboot_thread_fn+0xe2/0x1a0 [45759.026919] [] ? lg_double_unlock+0x40/0x40 [45759.033445] [] kthread+0xd1/0xe0 [45759.038921] [] ? insert_kthread_work+0x40/0x40 [45759.045746] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.053051] [] ? insert_kthread_work+0x40/0x40 [45759.059875] ksoftirqd/24 S ffff99e86a4fcb00 0 131 2 0x00000000 [45759.067821] Call Trace: [45759.070589] [] schedule+0x29/0x70 [45759.076156] [] smpboot_thread_fn+0xe2/0x1a0 [45759.082682] [] ? lg_double_unlock+0x40/0x40 [45759.089211] [] kthread+0xd1/0xe0 [45759.094686] [] ? insert_kthread_work+0x40/0x40 [45759.101510] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.108822] [] ? insert_kthread_work+0x40/0x40 [45759.115645] kworker/24:0H S ffff99e8f5ba2100 0 133 2 0x00000000 [45759.123631] Call Trace: [45759.126383] [] schedule+0x29/0x70 [45759.131957] [] worker_thread+0x1d9/0x3c0 [45759.138212] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45759.145512] [] kthread+0xd1/0xe0 [45759.150973] [] ? insert_kthread_work+0x40/0x40 [45759.157785] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.165096] [] ? insert_kthread_work+0x40/0x40 [45759.171918] watchdog/25 S ffff99e8f5ba3180 0 134 2 0x00000000 [45759.179866] Call Trace: [45759.182629] [] schedule+0x29/0x70 [45759.188212] [] smpboot_thread_fn+0xe2/0x1a0 [45759.194748] [] ? lg_double_unlock+0x40/0x40 [45759.201284] [] kthread+0xd1/0xe0 [45759.206758] [] ? insert_kthread_work+0x40/0x40 [45759.213583] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.220894] [] ? insert_kthread_work+0x40/0x40 [45759.227716] migration/25 S ffff99e8f5ba4200 0 135 2 0x00000000 [45759.235657] Call Trace: [45759.238410] [] schedule+0x29/0x70 [45759.243976] [] smpboot_thread_fn+0xe2/0x1a0 [45759.250510] [] ? lg_double_unlock+0x40/0x40 [45759.257042] [] kthread+0xd1/0xe0 [45759.262505] [] ? insert_kthread_work+0x40/0x40 [45759.269323] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.276636] [] ? insert_kthread_work+0x40/0x40 [45759.283472] ksoftirqd/25 S ffff99ec69696400 0 136 2 0x00000000 [45759.291412] Call Trace: [45759.294163] [] schedule+0x29/0x70 [45759.299728] [] smpboot_thread_fn+0xe2/0x1a0 [45759.306253] [] ? lg_double_unlock+0x40/0x40 [45759.312775] [] kthread+0xd1/0xe0 [45759.318250] [] ? insert_kthread_work+0x40/0x40 [45759.325086] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.332390] [] ? insert_kthread_work+0x40/0x40 [45759.339213] kworker/25:0H S ffff99e8f5410000 0 138 2 0x00000000 [45759.347200] Call Trace: [45759.349967] [] schedule+0x29/0x70 [45759.355536] [] worker_thread+0x1d9/0x3c0 [45759.361783] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45759.369099] [] kthread+0xd1/0xe0 [45759.374569] [] ? insert_kthread_work+0x40/0x40 [45759.381403] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.388716] [] ? insert_kthread_work+0x40/0x40 [45759.395538] watchdog/26 S ffff99e8f5411080 0 139 2 0x00000000 [45759.403483] Call Trace: [45759.406234] [] schedule+0x29/0x70 [45759.411799] [] smpboot_thread_fn+0xe2/0x1a0 [45759.418327] [] ? lg_double_unlock+0x40/0x40 [45759.424860] [] kthread+0xd1/0xe0 [45759.430333] [] ? insert_kthread_work+0x40/0x40 [45759.437163] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.444467] [] ? insert_kthread_work+0x40/0x40 [45759.451292] migration/26 S ffff99e8f5412100 0 140 2 0x00000000 [45759.459238] Call Trace: [45759.461991] [] schedule+0x29/0x70 [45759.467558] [] smpboot_thread_fn+0xe2/0x1a0 [45759.474087] [] ? lg_double_unlock+0x40/0x40 [45759.480622] [] kthread+0xd1/0xe0 [45759.486098] [] ? insert_kthread_work+0x40/0x40 [45759.492918] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.500232] [] ? insert_kthread_work+0x40/0x40 [45759.507071] ksoftirqd/26 S ffff99ec6b507080 0 141 2 0x00000000 [45759.515020] Call Trace: [45759.517766] [] schedule+0x29/0x70 [45759.523317] [] smpboot_thread_fn+0xe2/0x1a0 [45759.529842] [] ? lg_double_unlock+0x40/0x40 [45759.536369] [] kthread+0xd1/0xe0 [45759.541836] [] ? insert_kthread_work+0x40/0x40 [45759.548662] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.555971] [] ? insert_kthread_work+0x40/0x40 [45759.562789] kworker/26:0H S ffff99e8f5415280 0 143 2 0x00000000 [45759.570732] Call Trace: [45759.573470] [] schedule+0x29/0x70 [45759.579042] [] worker_thread+0x1d9/0x3c0 [45759.585270] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45759.592574] [] kthread+0xd1/0xe0 [45759.598022] [] ? insert_kthread_work+0x40/0x40 [45759.604841] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.612156] [] ? insert_kthread_work+0x40/0x40 [45759.618969] watchdog/27 S ffff99e8f5416300 0 144 2 0x00000000 [45759.626924] Call Trace: [45759.629678] [] schedule+0x29/0x70 [45759.635234] [] smpboot_thread_fn+0xe2/0x1a0 [45759.641768] [] ? lg_double_unlock+0x40/0x40 [45759.648288] [] kthread+0xd1/0xe0 [45759.653755] [] ? insert_kthread_work+0x40/0x40 [45759.660575] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.667874] [] ? insert_kthread_work+0x40/0x40 [45759.674691] migration/27 S ffff99e8f5478000 0 145 2 0x00000000 [45759.682621] Call Trace: [45759.685358] [] schedule+0x29/0x70 [45759.690908] [] smpboot_thread_fn+0xe2/0x1a0 [45759.697441] [] ? lg_double_unlock+0x40/0x40 [45759.703983] [] kthread+0xd1/0xe0 [45759.709436] [] ? insert_kthread_work+0x40/0x40 [45759.716255] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.723564] [] ? insert_kthread_work+0x40/0x40 [45759.730391] ksoftirqd/27 S ffff99e8f5479080 0 146 2 0x00000000 [45759.738301] Call Trace: [45759.741060] [] schedule+0x29/0x70 [45759.746619] [] smpboot_thread_fn+0xe2/0x1a0 [45759.753162] [] ? lg_double_unlock+0x40/0x40 [45759.759695] [] kthread+0xd1/0xe0 [45759.765160] [] ? insert_kthread_work+0x40/0x40 [45759.771984] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.779285] [] ? insert_kthread_work+0x40/0x40 [45759.786108] kworker/27:0H S ffff99e8f547b180 0 148 2 0x00000000 [45759.794048] Call Trace: [45759.796807] [] schedule+0x29/0x70 [45759.802360] [] worker_thread+0x1d9/0x3c0 [45759.808663] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45759.815974] [] kthread+0xd1/0xe0 [45759.821436] [] ? insert_kthread_work+0x40/0x40 [45759.828246] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.835566] [] ? insert_kthread_work+0x40/0x40 [45759.842404] watchdog/28 S ffff99e8f547c200 0 149 2 0x00000000 [45759.850333] Call Trace: [45759.853085] [] schedule+0x29/0x70 [45759.858642] [] smpboot_thread_fn+0xe2/0x1a0 [45759.865179] [] ? lg_double_unlock+0x40/0x40 [45759.871705] [] kthread+0xd1/0xe0 [45759.877180] [] ? insert_kthread_work+0x40/0x40 [45759.884006] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.891302] [] ? insert_kthread_work+0x40/0x40 [45759.898127] migration/28 S ffff99e8f547d280 0 150 2 0x00000000 [45759.906062] Call Trace: [45759.908815] [] schedule+0x29/0x70 [45759.914373] [] smpboot_thread_fn+0xe2/0x1a0 [45759.920900] [] ? lg_double_unlock+0x40/0x40 [45759.927418] [] kthread+0xd1/0xe0 [45759.932880] [] ? insert_kthread_work+0x40/0x40 [45759.939690] [] ret_from_fork_nospec_begin+0x21/0x21 [45759.947010] [] ? insert_kthread_work+0x40/0x40 [45759.953839] ksoftirqd/28 S ffff99e8f547e300 0 151 2 0x00000000 [45759.961801] Call Trace: [45759.964578] [] schedule+0x29/0x70 [45759.970170] [] smpboot_thread_fn+0xe2/0x1a0 [45759.976704] [] ? lg_double_unlock+0x40/0x40 [45759.983252] [] kthread+0xd1/0xe0 [45759.988727] [] ? insert_kthread_work+0x40/0x40 [45759.995554] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.002860] [] ? insert_kthread_work+0x40/0x40 [45760.009681] kworker/28:0 S ffff99e8f54c8000 0 152 2 0x00000000 [45760.017635] Call Trace: [45760.020397] [] schedule+0x29/0x70 [45760.025962] [] worker_thread+0x1d9/0x3c0 [45760.032205] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45760.039513] [] kthread+0xd1/0xe0 [45760.044967] [] ? insert_kthread_work+0x40/0x40 [45760.051777] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.059075] [] ? insert_kthread_work+0x40/0x40 [45760.065913] kworker/28:0H S ffff99e8f54c9080 0 153 2 0x00000000 [45760.073857] Call Trace: [45760.076615] [] schedule+0x29/0x70 [45760.082193] [] worker_thread+0x1d9/0x3c0 [45760.088429] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45760.095728] [] kthread+0xd1/0xe0 [45760.101183] [] ? insert_kthread_work+0x40/0x40 [45760.107987] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.115296] [] ? insert_kthread_work+0x40/0x40 [45760.122107] watchdog/29 S ffff99e8f54ca100 0 154 2 0x00000000 [45760.130030] Call Trace: [45760.132768] [] schedule+0x29/0x70 [45760.138321] [] smpboot_thread_fn+0xe2/0x1a0 [45760.144865] [] ? lg_double_unlock+0x40/0x40 [45760.151411] [] kthread+0xd1/0xe0 [45760.156901] [] ? insert_kthread_work+0x40/0x40 [45760.163735] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.171046] [] ? insert_kthread_work+0x40/0x40 [45760.177881] migration/29 S ffff99e8f54cb180 0 155 2 0x00000000 [45760.185840] Call Trace: [45760.188581] [] schedule+0x29/0x70 [45760.194135] [] smpboot_thread_fn+0xe2/0x1a0 [45760.200655] [] ? lg_double_unlock+0x40/0x40 [45760.207180] [] kthread+0xd1/0xe0 [45760.212635] [] ? insert_kthread_work+0x40/0x40 [45760.219437] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.226737] [] ? insert_kthread_work+0x40/0x40 [45760.233549] ksoftirqd/29 S ffff99e8f54cc200 0 156 2 0x00000000 [45760.241457] Call Trace: [45760.244195] [] schedule+0x29/0x70 [45760.249746] [] smpboot_thread_fn+0xe2/0x1a0 [45760.256259] [] ? lg_double_unlock+0x40/0x40 [45760.262770] [] kthread+0xd1/0xe0 [45760.268223] [] ? insert_kthread_work+0x40/0x40 [45760.275048] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.282334] [] ? insert_kthread_work+0x40/0x40 [45760.289145] kworker/29:0H S ffff99e8f54ce300 0 158 2 0x00000000 [45760.297057] Call Trace: [45760.299798] [] schedule+0x29/0x70 [45760.305350] [] worker_thread+0x1d9/0x3c0 [45760.311586] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45760.318881] [] kthread+0xd1/0xe0 [45760.324336] [] ? insert_kthread_work+0x40/0x40 [45760.331150] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.338446] [] ? insert_kthread_work+0x40/0x40 [45760.345265] watchdog/30 S ffff99e8f5548000 0 159 2 0x00000000 [45760.353196] Call Trace: [45760.355951] [] schedule+0x29/0x70 [45760.361518] [] smpboot_thread_fn+0xe2/0x1a0 [45760.368058] [] ? lg_double_unlock+0x40/0x40 [45760.374579] [] kthread+0xd1/0xe0 [45760.380054] [] ? insert_kthread_work+0x40/0x40 [45760.386874] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.394193] [] ? insert_kthread_work+0x40/0x40 [45760.401002] migration/30 S ffff99e8f5549080 0 160 2 0x00000000 [45760.408908] Call Trace: [45760.411676] [] schedule+0x29/0x70 [45760.417232] [] smpboot_thread_fn+0xe2/0x1a0 [45760.423779] [] ? lg_double_unlock+0x40/0x40 [45760.430306] [] kthread+0xd1/0xe0 [45760.435767] [] ? insert_kthread_work+0x40/0x40 [45760.442577] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.449887] [] ? insert_kthread_work+0x40/0x40 [45760.456698] ksoftirqd/30 S ffff99e8f554a100 0 161 2 0x00000000 [45760.464604] Call Trace: [45760.467343] [] schedule+0x29/0x70 [45760.472886] [] smpboot_thread_fn+0xe2/0x1a0 [45760.479398] [] ? lg_double_unlock+0x40/0x40 [45760.485919] [] kthread+0xd1/0xe0 [45760.491364] [] ? insert_kthread_work+0x40/0x40 [45760.498197] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.505493] [] ? insert_kthread_work+0x40/0x40 [45760.512297] kworker/30:0H S ffff99e8f554c200 0 163 2 0x00000000 [45760.520256] Call Trace: [45760.523021] [] schedule+0x29/0x70 [45760.528592] [] worker_thread+0x1d9/0x3c0 [45760.534851] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45760.542190] [] kthread+0xd1/0xe0 [45760.547651] [] ? insert_kthread_work+0x40/0x40 [45760.554469] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.561782] [] ? insert_kthread_work+0x40/0x40 [45760.568619] watchdog/31 S ffff99e8f554d280 0 164 2 0x00000000 [45760.576571] Call Trace: [45760.579325] [] schedule+0x29/0x70 [45760.584887] [] smpboot_thread_fn+0xe2/0x1a0 [45760.591423] [] ? lg_double_unlock+0x40/0x40 [45760.597956] [] kthread+0xd1/0xe0 [45760.603437] [] ? insert_kthread_work+0x40/0x40 [45760.610278] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.617580] [] ? insert_kthread_work+0x40/0x40 [45760.624421] migration/31 S ffff99e8f554e300 0 165 2 0x00000000 [45760.632349] Call Trace: [45760.635110] [] schedule+0x29/0x70 [45760.640664] [] smpboot_thread_fn+0xe2/0x1a0 [45760.647220] [] ? lg_double_unlock+0x40/0x40 [45760.653745] [] kthread+0xd1/0xe0 [45760.659238] [] ? insert_kthread_work+0x40/0x40 [45760.666085] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.673388] [] ? insert_kthread_work+0x40/0x40 [45760.680234] ksoftirqd/31 S ffff99e8f5590000 0 166 2 0x00000000 [45760.688202] Call Trace: [45760.690961] [] schedule+0x29/0x70 [45760.696519] [] smpboot_thread_fn+0xe2/0x1a0 [45760.703060] [] ? lg_double_unlock+0x40/0x40 [45760.709594] [] kthread+0xd1/0xe0 [45760.715076] [] ? insert_kthread_work+0x40/0x40 [45760.721903] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.729232] [] ? insert_kthread_work+0x40/0x40 [45760.736069] kworker/31:0H S ffff99e8f5592100 0 168 2 0x00000000 [45760.744056] Call Trace: [45760.746819] [] schedule+0x29/0x70 [45760.752385] [] worker_thread+0x1d9/0x3c0 [45760.758629] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45760.765947] [] kthread+0xd1/0xe0 [45760.771409] [] ? insert_kthread_work+0x40/0x40 [45760.778230] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.785547] [] ? insert_kthread_work+0x40/0x40 [45760.792363] kdevtmpfs S ffff99e5b504c200 0 171 2 0x00000000 [45760.800321] Call Trace: [45760.803075] [] schedule+0x29/0x70 [45760.808647] [] devtmpfsd+0x16d/0x180 [45760.814576] [] ? handle_create.isra.2+0x220/0x220 [45760.821696] [] kthread+0xd1/0xe0 [45760.827165] [] ? insert_kthread_work+0x40/0x40 [45760.833985] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.841295] [] ? insert_kthread_work+0x40/0x40 [45760.848132] netns S ffff99e5b504b760 0 172 2 0x00000000 [45760.856135] Call Trace: [45760.858888] [] schedule+0x29/0x70 [45760.864454] [] rescuer_thread+0x31e/0x3c0 [45760.870801] [] ? __schedule+0x320/0x680 [45760.876957] [] ? process_one_work+0x440/0x440 [45760.883686] [] kthread+0xd1/0xe0 [45760.889155] [] ? insert_kthread_work+0x40/0x40 [45760.895974] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.903290] [] ? insert_kthread_work+0x40/0x40 [45760.910120] khungtaskd S ffff99e5b504a100 0 173 2 0x00000000 [45760.918097] Call Trace: [45760.920842] [] schedule+0x29/0x70 [45760.926400] [] schedule_timeout+0x168/0x2d0 [45760.932928] [] ? requeue_timers+0x170/0x170 [45760.939460] [] schedule_timeout_interruptible+0x1e/0x20 [45760.947162] [] watchdog+0x4c/0x2c0 [45760.952817] [] ? reset_hung_task_detector+0x20/0x20 [45760.960134] [] kthread+0xd1/0xe0 [45760.965602] [] ? insert_kthread_work+0x40/0x40 [45760.972429] [] ret_from_fork_nospec_begin+0x21/0x21 [45760.979752] [] ? insert_kthread_work+0x40/0x40 [45760.986566] writeback S ffff99e86fb285e0 0 174 2 0x00000000 [45760.994512] Call Trace: [45760.997285] [] schedule+0x29/0x70 [45761.002852] [] rescuer_thread+0x31e/0x3c0 [45761.009217] [] ? __schedule+0x320/0x680 [45761.015363] [] ? process_one_work+0x440/0x440 [45761.022096] [] kthread+0xd1/0xe0 [45761.027541] [] ? insert_kthread_work+0x40/0x40 [45761.034360] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.041677] [] ? insert_kthread_work+0x40/0x40 [45761.048499] kintegrityd S ffff99e86fb29660 0 175 2 0x00000000 [45761.056453] Call Trace: [45761.059225] [] schedule+0x29/0x70 [45761.064798] [] rescuer_thread+0x31e/0x3c0 [45761.071141] [] ? __schedule+0x320/0x680 [45761.077300] [] ? process_one_work+0x440/0x440 [45761.084032] [] kthread+0xd1/0xe0 [45761.089491] [] ? insert_kthread_work+0x40/0x40 [45761.096310] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.103613] [] ? insert_kthread_work+0x40/0x40 [45761.110443] bioset S ffff99e86fb2a6e0 0 176 2 0x00000000 [45761.118398] Call Trace: [45761.121152] [] schedule+0x29/0x70 [45761.126709] [] rescuer_thread+0x31e/0x3c0 [45761.133057] [] ? __schedule+0x320/0x680 [45761.139224] [] ? process_one_work+0x440/0x440 [45761.145945] [] kthread+0xd1/0xe0 [45761.151410] [] ? insert_kthread_work+0x40/0x40 [45761.158248] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.165557] [] ? insert_kthread_work+0x40/0x40 [45761.172387] bioset S ffff99e86fb2b760 0 177 2 0x00000000 [45761.180372] Call Trace: [45761.183133] [] schedule+0x29/0x70 [45761.188698] [] rescuer_thread+0x31e/0x3c0 [45761.195047] [] ? __schedule+0x320/0x680 [45761.201188] [] ? process_one_work+0x440/0x440 [45761.207917] [] kthread+0xd1/0xe0 [45761.213384] [] ? insert_kthread_work+0x40/0x40 [45761.220236] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.227539] [] ? insert_kthread_work+0x40/0x40 [45761.234356] bioset S ffff99e86fb2c7e0 0 178 2 0x00000000 [45761.242301] Call Trace: [45761.245055] [] schedule+0x29/0x70 [45761.250621] [] rescuer_thread+0x31e/0x3c0 [45761.256970] [] ? __schedule+0x320/0x680 [45761.263125] [] ? process_one_work+0x440/0x440 [45761.269848] [] kthread+0xd1/0xe0 [45761.275316] [] ? insert_kthread_work+0x40/0x40 [45761.282143] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.289454] [] ? insert_kthread_work+0x40/0x40 [45761.296269] kblockd S ffff99e86fb2d860 0 179 2 0x00000000 [45761.304247] Call Trace: [45761.306993] [] schedule+0x29/0x70 [45761.312550] [] rescuer_thread+0x31e/0x3c0 [45761.318898] [] ? __schedule+0x320/0x680 [45761.325043] [] ? process_one_work+0x440/0x440 [45761.331771] [] kthread+0xd1/0xe0 [45761.337250] [] ? insert_kthread_work+0x40/0x40 [45761.344074] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.351391] [] ? insert_kthread_work+0x40/0x40 [45761.358237] md S ffff99e86f0b85e0 0 181 2 0x00000000 [45761.366243] Call Trace: [45761.369003] [] schedule+0x29/0x70 [45761.374574] [] rescuer_thread+0x31e/0x3c0 [45761.380922] [] ? __schedule+0x320/0x680 [45761.387077] [] ? process_one_work+0x440/0x440 [45761.393804] [] kthread+0xd1/0xe0 [45761.399274] [] ? insert_kthread_work+0x40/0x40 [45761.406119] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.413430] [] ? insert_kthread_work+0x40/0x40 [45761.420256] edac-poller S ffff99e86f0b9660 0 182 2 0x00000000 [45761.428201] Call Trace: [45761.430955] [] schedule+0x29/0x70 [45761.436525] [] rescuer_thread+0x31e/0x3c0 [45761.442867] [] ? __schedule+0x320/0x680 [45761.449020] [] ? process_one_work+0x440/0x440 [45761.455756] [] kthread+0xd1/0xe0 [45761.461243] [] ? insert_kthread_work+0x40/0x40 [45761.468077] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.475389] [] ? insert_kthread_work+0x40/0x40 [45761.482206] watchdogd S ffff99e86f0ba6e0 0 183 2 0x00000000 [45761.490181] Call Trace: [45761.492916] [] schedule+0x29/0x70 [45761.498466] [] rescuer_thread+0x31e/0x3c0 [45761.504805] [] ? __schedule+0x320/0x680 [45761.510953] [] ? process_one_work+0x440/0x440 [45761.517676] [] kthread+0xd1/0xe0 [45761.523139] [] ? insert_kthread_work+0x40/0x40 [45761.529959] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.537268] [] ? insert_kthread_work+0x40/0x40 [45761.544091] kworker/18:1 S ffff99e86f0bc200 0 185 2 0x00000000 [45761.552059] Call Trace: [45761.554808] [] schedule+0x29/0x70 [45761.560388] [] worker_thread+0x1d9/0x3c0 [45761.566639] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45761.573949] [] kthread+0xd1/0xe0 [45761.579419] [] ? insert_kthread_work+0x40/0x40 [45761.586265] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.593574] [] ? insert_kthread_work+0x40/0x40 [45761.600390] kworker/17:1 S ffff99e86f0be300 0 187 2 0x00000000 [45761.608331] Call Trace: [45761.611077] [] schedule+0x29/0x70 [45761.616653] [] worker_thread+0x1d9/0x3c0 [45761.622912] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45761.630246] [] kthread+0xd1/0xe0 [45761.635715] [] ? insert_kthread_work+0x40/0x40 [45761.642533] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.649836] [] ? insert_kthread_work+0x40/0x40 [45761.656658] kswapd0 S ffff99e86d0b47e0 0 192 2 0x00000000 [45761.664599] Call Trace: [45761.667347] [] schedule+0x29/0x70 [45761.672896] [] kswapd+0x376/0x430 [45761.678461] [] ? wake_up_atomic_t+0x30/0x30 [45761.685001] [] ? balance_pgdat+0x5e0/0x5e0 [45761.691445] [] kthread+0xd1/0xe0 [45761.696900] [] ? insert_kthread_work+0x40/0x40 [45761.703718] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.711027] [] ? insert_kthread_work+0x40/0x40 [45761.717850] kswapd1 S ffff99e86d0b5860 0 193 2 0x00000000 [45761.725787] Call Trace: [45761.728533] [] schedule+0x29/0x70 [45761.734096] [] kswapd+0x376/0x430 [45761.739661] [] ? wake_up_atomic_t+0x30/0x30 [45761.746194] [] ? balance_pgdat+0x5e0/0x5e0 [45761.752626] [] kthread+0xd1/0xe0 [45761.758102] [] ? insert_kthread_work+0x40/0x40 [45761.764929] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.772254] [] ? insert_kthread_work+0x40/0x40 [45761.779069] ksmd S ffff99e86d0b68e0 0 194 2 0x00000000 [45761.787013] Call Trace: [45761.789768] [] schedule+0x29/0x70 [45761.795336] [] ksm_scan_thread+0x16a/0x240 [45761.801766] [] ? wake_up_atomic_t+0x30/0x30 [45761.808319] [] ? ksm_do_scan+0x11c0/0x11c0 [45761.814750] [] kthread+0xd1/0xe0 [45761.820246] [] ? insert_kthread_work+0x40/0x40 [45761.827066] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.834378] [] ? insert_kthread_work+0x40/0x40 [45761.841203] khugepaged S ffff99e86d0b3180 0 195 2 0x00000000 [45761.849150] Call Trace: [45761.851896] [] schedule+0x29/0x70 [45761.857454] [] schedule_timeout+0x168/0x2d0 [45761.863981] [] ? requeue_timers+0x170/0x170 [45761.870517] [] khugepaged+0x226/0x490 [45761.876469] [] ? wake_up_atomic_t+0x30/0x30 [45761.883004] [] ? khugepaged_scan_mm_slot+0xfd0/0xfd0 [45761.890411] [] kthread+0xd1/0xe0 [45761.895872] [] ? insert_kthread_work+0x40/0x40 [45761.902691] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.910004] [] ? insert_kthread_work+0x40/0x40 [45761.916824] crypto S ffff99e86d0b26e0 0 196 2 0x00000000 [45761.924778] Call Trace: [45761.927516] [] schedule+0x29/0x70 [45761.933080] [] rescuer_thread+0x31e/0x3c0 [45761.939427] [] ? __schedule+0x320/0x680 [45761.945581] [] ? process_one_work+0x440/0x440 [45761.952321] [] kthread+0xd1/0xe0 [45761.957780] [] ? insert_kthread_work+0x40/0x40 [45761.964607] [] ret_from_fork_nospec_begin+0x21/0x21 [45761.971918] [] ? insert_kthread_work+0x40/0x40 [45761.978734] kthrotld S ffff99e86d33d860 0 204 2 0x00000000 [45761.986686] Call Trace: [45761.989438] [] schedule+0x29/0x70 [45761.994998] [] rescuer_thread+0x31e/0x3c0 [45762.001354] [] ? __schedule+0x320/0x680 [45762.007486] [] ? process_one_work+0x440/0x440 [45762.014224] [] kthread+0xd1/0xe0 [45762.019693] [] ? insert_kthread_work+0x40/0x40 [45762.026512] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.033822] [] ? insert_kthread_work+0x40/0x40 [45762.040654] kmpath_rdacd S ffff99ec6bd4d860 0 209 2 0x00000000 [45762.048605] Call Trace: [45762.051365] [] schedule+0x29/0x70 [45762.056931] [] rescuer_thread+0x31e/0x3c0 [45762.063286] [] ? __schedule+0x320/0x680 [45762.069426] [] ? process_one_work+0x440/0x440 [45762.076160] [] kthread+0xd1/0xe0 [45762.081627] [] ? insert_kthread_work+0x40/0x40 [45762.088446] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.095755] [] ? insert_kthread_work+0x40/0x40 [45762.102565] kaluad S ffff99ec6bd4c7e0 0 210 2 0x00000000 [45762.110510] Call Trace: [45762.113276] [] schedule+0x29/0x70 [45762.118842] [] rescuer_thread+0x31e/0x3c0 [45762.125183] [] ? __schedule+0x320/0x680 [45762.131333] [] ? process_one_work+0x440/0x440 [45762.138054] [] kthread+0xd1/0xe0 [45762.143529] [] ? insert_kthread_work+0x40/0x40 [45762.150354] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.157662] [] ? insert_kthread_work+0x40/0x40 [45762.164490] kpsmoused S ffff99ec6bd4a6e0 0 213 2 0x00000000 [45762.172424] Call Trace: [45762.175180] [] schedule+0x29/0x70 [45762.180755] [] rescuer_thread+0x31e/0x3c0 [45762.187102] [] ? __schedule+0x320/0x680 [45762.193271] [] ? process_one_work+0x440/0x440 [45762.199997] [] kthread+0xd1/0xe0 [45762.205473] [] ? insert_kthread_work+0x40/0x40 [45762.212296] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.219607] [] ? insert_kthread_work+0x40/0x40 [45762.226431] ipv6_addrconf S ffff99ec6bd4b760 0 214 2 0x00000000 [45762.234375] Call Trace: [45762.237134] [] schedule+0x29/0x70 [45762.242694] [] rescuer_thread+0x31e/0x3c0 [45762.249025] [] ? __schedule+0x320/0x680 [45762.255171] [] ? process_one_work+0x440/0x440 [45762.261905] [] kthread+0xd1/0xe0 [45762.267366] [] ? insert_kthread_work+0x40/0x40 [45762.274190] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.281501] [] ? insert_kthread_work+0x40/0x40 [45762.288337] kworker/10:1 S ffff99ec6b958000 0 215 2 0x00000000 [45762.296335] Call Trace: [45762.299087] [] schedule+0x29/0x70 [45762.304653] [] worker_thread+0x1d9/0x3c0 [45762.310898] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.318219] [] kthread+0xd1/0xe0 [45762.323694] [] ? insert_kthread_work+0x40/0x40 [45762.330523] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.337834] [] ? insert_kthread_work+0x40/0x40 [45762.344664] deferwq S ffff99ec6b95d860 0 228 2 0x00000000 [45762.352593] Call Trace: [45762.355357] [] schedule+0x29/0x70 [45762.360924] [] rescuer_thread+0x31e/0x3c0 [45762.367280] [] ? __schedule+0x320/0x680 [45762.373427] [] ? process_one_work+0x440/0x440 [45762.380155] [] kthread+0xd1/0xe0 [45762.385624] [] ? insert_kthread_work+0x40/0x40 [45762.392441] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.399764] [] ? insert_kthread_work+0x40/0x40 [45762.406584] kworker/5:1 S ffff99e86d339080 0 233 2 0x00000000 [45762.414531] Call Trace: [45762.417292] [] schedule+0x29/0x70 [45762.422857] [] worker_thread+0x1d9/0x3c0 [45762.429102] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.436404] [] kthread+0xd1/0xe0 [45762.441874] [] ? insert_kthread_work+0x40/0x40 [45762.448708] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.456025] [] ? insert_kthread_work+0x40/0x40 [45762.462856] kworker/7:1 S ffff99e86cf80000 0 234 2 0x00000000 [45762.470800] Call Trace: [45762.473561] [] schedule+0x29/0x70 [45762.479127] [] worker_thread+0x1d9/0x3c0 [45762.485373] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.492669] [] kthread+0xd1/0xe0 [45762.498130] [] ? insert_kthread_work+0x40/0x40 [45762.504955] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.512287] [] ? insert_kthread_work+0x40/0x40 [45762.519117] kworker/16:1 S ffff99ec69696a40 0 239 2 0x00000000 [45762.527083] Call Trace: [45762.529835] [] schedule+0x29/0x70 [45762.535401] [] worker_thread+0x1d9/0x3c0 [45762.541644] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.548954] [] kthread+0xd1/0xe0 [45762.554417] [] ? insert_kthread_work+0x40/0x40 [45762.561248] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.568574] [] ? insert_kthread_work+0x40/0x40 [45762.575399] kworker/19:1 S ffff99e86cf85280 0 240 2 0x00000000 [45762.583349] Call Trace: [45762.586104] [] schedule+0x29/0x70 [45762.591676] [] worker_thread+0x1d9/0x3c0 [45762.597920] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.605232] [] kthread+0xd1/0xe0 [45762.610692] [] ? insert_kthread_work+0x40/0x40 [45762.617506] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.624825] [] ? insert_kthread_work+0x40/0x40 [45762.631637] kworker/20:1 S ffff99e86cf84200 0 241 2 0x00000000 [45762.639563] Call Trace: [45762.642302] [] schedule+0x29/0x70 [45762.647880] [] worker_thread+0x1d9/0x3c0 [45762.654139] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.661448] [] kthread+0xd1/0xe0 [45762.666924] [] ? insert_kthread_work+0x40/0x40 [45762.673752] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.681063] [] ? insert_kthread_work+0x40/0x40 [45762.687895] kworker/23:1 S ffff99e86cf82100 0 243 2 0x00000000 [45762.695843] Call Trace: [45762.698591] [] schedule+0x29/0x70 [45762.704158] [] worker_thread+0x1d9/0x3c0 [45762.710408] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.717724] [] kthread+0xd1/0xe0 [45762.723192] [] ? insert_kthread_work+0x40/0x40 [45762.730018] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.737314] [] ? insert_kthread_work+0x40/0x40 [45762.744146] kworker/22:1 S ffff99e86cf81080 0 244 2 0x00000000 [45762.752104] Call Trace: [45762.754852] [] schedule+0x29/0x70 [45762.760416] [] worker_thread+0x1d9/0x3c0 [45762.766661] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.773977] [] kthread+0xd1/0xe0 [45762.779436] [] ? insert_kthread_work+0x40/0x40 [45762.786264] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.793590] [] ? insert_kthread_work+0x40/0x40 [45762.800420] kworker/24:1 S ffff99ec6ba76300 0 245 2 0x00000000 [45762.808375] Call Trace: [45762.811127] [] schedule+0x29/0x70 [45762.816694] [] worker_thread+0x1d9/0x3c0 [45762.822947] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.830419] [] kthread+0xd1/0xe0 [45762.835905] [] ? insert_kthread_work+0x40/0x40 [45762.842721] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.850037] [] ? insert_kthread_work+0x40/0x40 [45762.856874] kworker/28:1 S ffff99ec6ba74200 0 247 2 0x00000000 [45762.864815] Call Trace: [45762.867563] [] schedule+0x29/0x70 [45762.873131] [] worker_thread+0x1d9/0x3c0 [45762.879387] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.886689] [] kthread+0xd1/0xe0 [45762.892166] [] ? insert_kthread_work+0x40/0x40 [45762.898982] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.906317] [] ? insert_kthread_work+0x40/0x40 [45762.913148] kworker/29:1 S ffff99ec6ba90000 0 248 2 0x00000000 [45762.921109] Call Trace: [45762.923863] [] schedule+0x29/0x70 [45762.929429] [] worker_thread+0x1d9/0x3c0 [45762.935673] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.942992] [] kthread+0xd1/0xe0 [45762.948453] [] ? insert_kthread_work+0x40/0x40 [45762.955300] [] ret_from_fork_nospec_begin+0x21/0x21 [45762.962597] [] ? insert_kthread_work+0x40/0x40 [45762.969420] kworker/27:1 S ffff99ec6ba91080 0 249 2 0x00000000 [45762.977346] Call Trace: [45762.980112] [] schedule+0x29/0x70 [45762.985677] [] worker_thread+0x1d9/0x3c0 [45762.991928] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45762.999237] [] kthread+0xd1/0xe0 [45763.004714] [] ? insert_kthread_work+0x40/0x40 [45763.011533] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.018844] [] ? insert_kthread_work+0x40/0x40 [45763.025682] kworker/30:1 S ffff99ec6ba92100 0 250 2 0x00000000 [45763.033638] Call Trace: [45763.036393] [] schedule+0x29/0x70 [45763.041966] [] worker_thread+0x1d9/0x3c0 [45763.048212] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45763.055516] [] kthread+0xd1/0xe0 [45763.060992] [] ? insert_kthread_work+0x40/0x40 [45763.067819] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.075123] [] ? insert_kthread_work+0x40/0x40 [45763.081968] kworker/31:1 S ffff99ec6ba93180 0 251 2 0x00000000 [45763.089941] Call Trace: [45763.092695] [] schedule+0x29/0x70 [45763.098264] [] worker_thread+0x1d9/0x3c0 [45763.104501] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45763.111811] [] kthread+0xd1/0xe0 [45763.117268] [] ? insert_kthread_work+0x40/0x40 [45763.124094] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.131412] [] ? insert_kthread_work+0x40/0x40 [45763.138235] kworker/25:1 S ffff99ec6ba94200 0 252 2 0x00000000 [45763.146186] Call Trace: [45763.148939] [] schedule+0x29/0x70 [45763.154503] [] worker_thread+0x1d9/0x3c0 [45763.160754] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45763.168064] [] kthread+0xd1/0xe0 [45763.173534] [] ? insert_kthread_work+0x40/0x40 [45763.180377] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.187688] [] ? insert_kthread_work+0x40/0x40 [45763.194512] kworker/11:1 S ffff99ec6bab0000 0 254 2 0x00000000 [45763.202437] Call Trace: [45763.205182] [] schedule+0x29/0x70 [45763.210739] [] worker_thread+0x1d9/0x3c0 [45763.216991] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45763.224315] [] kthread+0xd1/0xe0 [45763.229786] [] ? insert_kthread_work+0x40/0x40 [45763.236604] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.243922] [] ? insert_kthread_work+0x40/0x40 [45763.250740] kauditd S ffff99ec6bab3180 0 340 2 0x00000000 [45763.258666] Call Trace: [45763.261418] [] schedule+0x29/0x70 [45763.266983] [] kauditd_thread+0x184/0x1e0 [45763.273335] [] ? wake_up_state+0x20/0x20 [45763.279579] [] ? audit_printk_skb+0x70/0x70 [45763.286119] [] kthread+0xd1/0xe0 [45763.291586] [] ? insert_kthread_work+0x40/0x40 [45763.298413] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.305724] [] ? insert_kthread_work+0x40/0x40 [45763.312542] ata_sff S ffff99ec6bab5860 0 464 2 0x00000000 [45763.320489] Call Trace: [45763.323234] [] schedule+0x29/0x70 [45763.328801] [] rescuer_thread+0x31e/0x3c0 [45763.335143] [] ? __schedule+0x320/0x680 [45763.341285] [] ? process_one_work+0x440/0x440 [45763.348022] [] kthread+0xd1/0xe0 [45763.353489] [] ? insert_kthread_work+0x40/0x40 [45763.360334] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.367643] [] ? insert_kthread_work+0x40/0x40 [45763.374467] scsi_eh_0 S ffff99e5b504d860 0 479 2 0x00000000 [45763.382394] Call Trace: [45763.385147] [] schedule+0x29/0x70 [45763.390712] [] scsi_error_handler+0xa3/0x8b0 [45763.397345] [] ? scsi_eh_get_sense+0x250/0x250 [45763.404171] [] kthread+0xd1/0xe0 [45763.409638] [] ? insert_kthread_work+0x40/0x40 [45763.416472] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.423788] [] ? insert_kthread_work+0x40/0x40 [45763.430612] scsi_tmf_0 S ffff99e86a670000 0 480 2 0x00000000 [45763.438552] Call Trace: [45763.441324] [] schedule+0x29/0x70 [45763.446888] [] rescuer_thread+0x31e/0x3c0 [45763.453230] [] ? __schedule+0x320/0x680 [45763.459378] [] ? process_one_work+0x440/0x440 [45763.466104] [] kthread+0xd1/0xe0 [45763.471572] [] ? insert_kthread_work+0x40/0x40 [45763.478412] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.485729] [] ? insert_kthread_work+0x40/0x40 [45763.492548] scsi_eh_1 S ffff99e86a671660 0 481 2 0x00000000 [45763.500491] Call Trace: [45763.503243] [] schedule+0x29/0x70 [45763.508806] [] scsi_error_handler+0xa3/0x8b0 [45763.515438] [] ? scsi_eh_get_sense+0x250/0x250 [45763.522248] [] kthread+0xd1/0xe0 [45763.527716] [] ? insert_kthread_work+0x40/0x40 [45763.534537] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.541840] [] ? insert_kthread_work+0x40/0x40 [45763.548671] scsi_tmf_1 S ffff99e86a6726e0 0 482 2 0x00000000 [45763.556626] Call Trace: [45763.559386] [] schedule+0x29/0x70 [45763.564976] [] rescuer_thread+0x31e/0x3c0 [45763.571321] [] ? __schedule+0x320/0x680 [45763.577456] [] ? process_one_work+0x440/0x440 [45763.584183] [] kthread+0xd1/0xe0 [45763.589644] [] ? insert_kthread_work+0x40/0x40 [45763.596469] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.603785] [] ? insert_kthread_work+0x40/0x40 [45763.610609] scsi_wq_1 S ffff99e86a673760 0 483 2 0x00000000 [45763.618562] Call Trace: [45763.621336] [] schedule+0x29/0x70 [45763.626894] [] rescuer_thread+0x31e/0x3c0 [45763.633236] [] ? __schedule+0x320/0x680 [45763.639394] [] ? process_one_work+0x440/0x440 [45763.646115] [] kthread+0xd1/0xe0 [45763.651582] [] ? insert_kthread_work+0x40/0x40 [45763.658417] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.665728] [] ? insert_kthread_work+0x40/0x40 [45763.672543] scsi_eh_2 S ffff99e86a6747e0 0 484 2 0x00000000 [45763.680497] Call Trace: [45763.683249] [] schedule+0x29/0x70 [45763.688812] [] scsi_error_handler+0xa3/0x8b0 [45763.695435] [] ? scsi_eh_get_sense+0x250/0x250 [45763.702245] [] kthread+0xd1/0xe0 [45763.707713] [] ? insert_kthread_work+0x40/0x40 [45763.714532] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.721849] [] ? insert_kthread_work+0x40/0x40 [45763.728685] scsi_tmf_2 S ffff99e86a675860 0 485 2 0x00000000 [45763.736649] Call Trace: [45763.739397] [] schedule+0x29/0x70 [45763.744954] [] rescuer_thread+0x31e/0x3c0 [45763.751298] [] ? __schedule+0x320/0x680 [45763.757432] [] ? process_one_work+0x440/0x440 [45763.764160] [] kthread+0xd1/0xe0 [45763.769621] [] ? insert_kthread_work+0x40/0x40 [45763.776448] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.783759] [] ? insert_kthread_work+0x40/0x40 [45763.790582] scsi_eh_3 S ffff99e86a676300 0 486 2 0x00000000 [45763.798547] Call Trace: [45763.801294] [] schedule+0x29/0x70 [45763.806858] [] scsi_error_handler+0xa3/0x8b0 [45763.813488] [] ? scsi_eh_get_sense+0x250/0x250 [45763.820301] [] kthread+0xd1/0xe0 [45763.825813] [] ? insert_kthread_work+0x40/0x40 [45763.832656] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.839987] [] ? insert_kthread_work+0x40/0x40 [45763.846802] scsi_tmf_3 S ffff99e86b3a05e0 0 487 2 0x00000000 [45763.854734] Call Trace: [45763.857487] [] schedule+0x29/0x70 [45763.863058] [] rescuer_thread+0x31e/0x3c0 [45763.869418] [] ? __schedule+0x320/0x680 [45763.875564] [] ? process_one_work+0x440/0x440 [45763.882289] [] kthread+0xd1/0xe0 [45763.887759] [] ? insert_kthread_work+0x40/0x40 [45763.894579] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.901887] [] ? insert_kthread_work+0x40/0x40 [45763.908715] scsi_eh_4 S ffff99e86b3a1660 0 488 2 0x00000000 [45763.916665] Call Trace: [45763.919422] [] schedule+0x29/0x70 [45763.925001] [] scsi_error_handler+0xa3/0x8b0 [45763.931640] [] ? scsi_eh_get_sense+0x250/0x250 [45763.938465] [] kthread+0xd1/0xe0 [45763.943933] [] ? insert_kthread_work+0x40/0x40 [45763.950760] [] ret_from_fork_nospec_begin+0x21/0x21 [45763.958070] [] ? insert_kthread_work+0x40/0x40 [45763.964902] scsi_tmf_4 S ffff99e86b3a26e0 0 489 2 0x00000000 [45763.972859] Call Trace: [45763.975613] [] schedule+0x29/0x70 [45763.981187] [] rescuer_thread+0x31e/0x3c0 [45763.987536] [] ? __schedule+0x320/0x680 [45763.993690] [] ? process_one_work+0x440/0x440 [45764.000436] [] kthread+0xd1/0xe0 [45764.005904] [] ? insert_kthread_work+0x40/0x40 [45764.012731] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.020049] [] ? insert_kthread_work+0x40/0x40 [45764.026867] scsi_eh_5 S ffff99e86b3a3760 0 490 2 0x00000000 [45764.034801] Call Trace: [45764.037539] [] schedule+0x29/0x70 [45764.043096] [] scsi_error_handler+0xa3/0x8b0 [45764.049733] [] ? scsi_eh_get_sense+0x250/0x250 [45764.056566] [] kthread+0xd1/0xe0 [45764.062042] [] ? insert_kthread_work+0x40/0x40 [45764.068868] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.076177] [] ? insert_kthread_work+0x40/0x40 [45764.082993] scsi_tmf_5 S ffff99e86b3a47e0 0 491 2 0x00000000 [45764.090934] Call Trace: [45764.093678] [] schedule+0x29/0x70 [45764.099243] [] rescuer_thread+0x31e/0x3c0 [45764.105577] [] ? __schedule+0x320/0x680 [45764.111723] [] ? process_one_work+0x440/0x440 [45764.118458] [] kthread+0xd1/0xe0 [45764.123940] [] ? insert_kthread_work+0x40/0x40 [45764.130766] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.138074] [] ? insert_kthread_work+0x40/0x40 [45764.144903] scsi_eh_6 S ffff99e86b3a5860 0 492 2 0x00000000 [45764.152844] Call Trace: [45764.155588] [] schedule+0x29/0x70 [45764.161153] [] scsi_error_handler+0xa3/0x8b0 [45764.167783] [] ? scsi_eh_get_sense+0x250/0x250 [45764.174609] [] kthread+0xd1/0xe0 [45764.180070] [] ? insert_kthread_work+0x40/0x40 [45764.186895] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.194205] [] ? insert_kthread_work+0x40/0x40 [45764.201027] scsi_tmf_6 S ffff99e86b3a68e0 0 493 2 0x00000000 [45764.208981] Call Trace: [45764.211720] [] schedule+0x29/0x70 [45764.217263] [] rescuer_thread+0x31e/0x3c0 [45764.223595] [] ? __schedule+0x320/0x680 [45764.229734] [] ? process_one_work+0x440/0x440 [45764.236462] [] kthread+0xd1/0xe0 [45764.241936] [] ? insert_kthread_work+0x40/0x40 [45764.248775] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.256083] [] ? insert_kthread_work+0x40/0x40 [45764.262905] scsi_eh_7 S ffff99e869dd05e0 0 494 2 0x00000000 [45764.270854] Call Trace: [45764.273602] [] schedule+0x29/0x70 [45764.279167] [] scsi_error_handler+0xa3/0x8b0 [45764.285799] [] ? scsi_eh_get_sense+0x250/0x250 [45764.292624] [] kthread+0xd1/0xe0 [45764.298092] [] ? insert_kthread_work+0x40/0x40 [45764.304925] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.312229] [] ? insert_kthread_work+0x40/0x40 [45764.319058] scsi_tmf_7 S ffff99e869dd1660 0 495 2 0x00000000 [45764.327001] Call Trace: [45764.329745] [] schedule+0x29/0x70 [45764.335312] [] rescuer_thread+0x31e/0x3c0 [45764.341653] [] ? __schedule+0x320/0x680 [45764.347790] [] ? process_one_work+0x440/0x440 [45764.354511] [] kthread+0xd1/0xe0 [45764.359984] [] ? insert_kthread_work+0x40/0x40 [45764.366815] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.374132] [] ? insert_kthread_work+0x40/0x40 [45764.380970] mlx4 S ffff99e869df9660 0 502 2 0x00000000 [45764.388931] Call Trace: [45764.391677] [] schedule+0x29/0x70 [45764.397250] [] rescuer_thread+0x31e/0x3c0 [45764.403594] [] ? __schedule+0x320/0x680 [45764.409741] [] ? process_one_work+0x440/0x440 [45764.416468] [] kthread+0xd1/0xe0 [45764.421936] [] ? insert_kthread_work+0x40/0x40 [45764.428761] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.436084] [] ? insert_kthread_work+0x40/0x40 [45764.442899] mlx4_health S ffff99e869dfa6e0 0 503 2 0x00000000 [45764.450847] Call Trace: [45764.453592] [] schedule+0x29/0x70 [45764.459149] [] rescuer_thread+0x31e/0x3c0 [45764.465488] [] ? __schedule+0x320/0x680 [45764.471633] [] ? process_one_work+0x440/0x440 [45764.478390] [] kthread+0xd1/0xe0 [45764.483862] [] ? insert_kthread_work+0x40/0x40 [45764.490677] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.497987] [] ? insert_kthread_work+0x40/0x40 [45764.504818] kworker/u305:8 S ffff99e869dfc200 0 505 2 0x00000000 [45764.512762] Call Trace: [45764.515515] [] schedule+0x29/0x70 [45764.521081] [] worker_thread+0x1d9/0x3c0 [45764.527311] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.534614] [] kthread+0xd1/0xe0 [45764.540090] [] ? insert_kthread_work+0x40/0x40 [45764.546908] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.554218] [] ? insert_kthread_work+0x40/0x40 [45764.561032] ttm_swap S ffff99e869dfd860 0 506 2 0x00000000 [45764.569028] Call Trace: [45764.571791] [] schedule+0x29/0x70 [45764.577344] [] rescuer_thread+0x31e/0x3c0 [45764.583705] [] ? __schedule+0x320/0x680 [45764.589853] [] ? process_one_work+0x440/0x440 [45764.596582] [] kthread+0xd1/0xe0 [45764.602036] [] ? insert_kthread_work+0x40/0x40 [45764.608860] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.616170] [] ? insert_kthread_work+0x40/0x40 [45764.622995] poll_mpt2sas0_s S ffff99e869dfe8e0 0 524 2 0x00000000 [45764.630958] Call Trace: [45764.633713] [] schedule+0x29/0x70 [45764.639291] [] rescuer_thread+0x31e/0x3c0 [45764.645631] [] ? __schedule+0x320/0x680 [45764.651779] [] ? process_one_work+0x440/0x440 [45764.658501] [] kthread+0xd1/0xe0 [45764.663972] [] ? insert_kthread_work+0x40/0x40 [45764.670789] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.678107] [] ? insert_kthread_work+0x40/0x40 [45764.684929] kworker/6:1H S ffff99e86d3e9080 0 529 2 0x00000000 [45764.692899] Call Trace: [45764.695653] [] schedule+0x29/0x70 [45764.701226] [] worker_thread+0x1d9/0x3c0 [45764.707471] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.714774] [] kthread+0xd1/0xe0 [45764.720234] [] ? insert_kthread_work+0x40/0x40 [45764.727047] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.734382] [] ? insert_kthread_work+0x40/0x40 [45764.741204] kworker/7:1H S ffff99e86d3ea100 0 530 2 0x00000000 [45764.749174] Call Trace: [45764.751927] [] schedule+0x29/0x70 [45764.757498] [] worker_thread+0x1d9/0x3c0 [45764.763730] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.771033] [] kthread+0xd1/0xe0 [45764.776493] [] ? insert_kthread_work+0x40/0x40 [45764.783323] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.790612] [] ? insert_kthread_work+0x40/0x40 [45764.797454] kworker/5:1H S ffff99e86d3eb180 0 531 2 0x00000000 [45764.805406] Call Trace: [45764.808160] [] schedule+0x29/0x70 [45764.813725] [] worker_thread+0x1d9/0x3c0 [45764.819982] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.827287] [] kthread+0xd1/0xe0 [45764.832815] [] ? insert_kthread_work+0x40/0x40 [45764.839643] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.846953] [] ? insert_kthread_work+0x40/0x40 [45764.853778] kworker/3:1H S ffff99e86d3ec200 0 532 2 0x00000000 [45764.861718] Call Trace: [45764.864468] [] schedule+0x29/0x70 [45764.870035] [] worker_thread+0x1d9/0x3c0 [45764.876278] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.883580] [] kthread+0xd1/0xe0 [45764.889062] [] ? insert_kthread_work+0x40/0x40 [45764.895896] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.903214] [] ? insert_kthread_work+0x40/0x40 [45764.910038] kworker/4:1H S ffff99e86d3ed280 0 533 2 0x00000000 [45764.917978] Call Trace: [45764.920724] [] schedule+0x29/0x70 [45764.926298] [] worker_thread+0x1d9/0x3c0 [45764.932544] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.939854] [] kthread+0xd1/0xe0 [45764.945331] [] ? insert_kthread_work+0x40/0x40 [45764.952144] [] ret_from_fork_nospec_begin+0x21/0x21 [45764.959475] [] ? insert_kthread_work+0x40/0x40 [45764.966304] kworker/17:1H S ffff99e86d3ee300 0 534 2 0x00000000 [45764.974270] Call Trace: [45764.977014] [] schedule+0x29/0x70 [45764.982581] [] worker_thread+0x1d9/0x3c0 [45764.988827] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45764.996136] [] kthread+0xd1/0xe0 [45765.001605] [] ? insert_kthread_work+0x40/0x40 [45765.008438] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.015757] [] ? insert_kthread_work+0x40/0x40 [45765.022593] ib-comp-wq S ffff99e84e7c1660 0 537 2 0x00000000 [45765.030552] Call Trace: [45765.033304] [] schedule+0x29/0x70 [45765.038861] [] rescuer_thread+0x31e/0x3c0 [45765.045200] [] ? __schedule+0x320/0x680 [45765.051342] [] ? process_one_work+0x440/0x440 [45765.058062] [] kthread+0xd1/0xe0 [45765.063530] [] ? insert_kthread_work+0x40/0x40 [45765.070356] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.077667] [] ? insert_kthread_work+0x40/0x40 [45765.084498] kworker/u307:0 S ffff99e84e7c26e0 0 538 2 0x00000000 [45765.092470] Call Trace: [45765.095214] [] schedule+0x29/0x70 [45765.100780] [] worker_thread+0x1d9/0x3c0 [45765.107016] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45765.114321] [] kthread+0xd1/0xe0 [45765.119789] [] ? insert_kthread_work+0x40/0x40 [45765.126617] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.133934] [] ? insert_kthread_work+0x40/0x40 [45765.140765] kworker/u308:0 S ffff99e84e7c3180 0 539 2 0x00000000 [45765.148735] Call Trace: [45765.151481] [] schedule+0x29/0x70 [45765.157033] [] worker_thread+0x1d9/0x3c0 [45765.163273] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45765.170575] [] kthread+0xd1/0xe0 [45765.176049] [] ? insert_kthread_work+0x40/0x40 [45765.182872] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.190188] [] ? insert_kthread_work+0x40/0x40 [45765.197011] kworker/u309:0 S ffff99ec6b95b760 0 540 2 0x00000000 [45765.204982] Call Trace: [45765.207727] [] schedule+0x29/0x70 [45765.213300] [] worker_thread+0x1d9/0x3c0 [45765.219544] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45765.226853] [] kthread+0xd1/0xe0 [45765.232308] [] ? insert_kthread_work+0x40/0x40 [45765.239127] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.246443] [] ? insert_kthread_work+0x40/0x40 [45765.253283] ib-comp-unb-wq S ffff99ec6b959660 0 541 2 0x00000000 [45765.261231] Call Trace: [45765.263978] [] schedule+0x29/0x70 [45765.269548] [] rescuer_thread+0x31e/0x3c0 [45765.275895] [] ? __schedule+0x320/0x680 [45765.282040] [] ? process_one_work+0x440/0x440 [45765.288767] [] kthread+0xd1/0xe0 [45765.294228] [] ? insert_kthread_work+0x40/0x40 [45765.301054] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.308365] [] ? insert_kthread_work+0x40/0x40 [45765.315196] ib_mcast S ffff99ec6b95c7e0 0 542 2 0x00000000 [45765.323142] Call Trace: [45765.325887] [] schedule+0x29/0x70 [45765.331480] [] rescuer_thread+0x31e/0x3c0 [45765.337827] [] ? __schedule+0x320/0x680 [45765.343966] [] ? process_one_work+0x440/0x440 [45765.350695] [] kthread+0xd1/0xe0 [45765.356163] [] ? insert_kthread_work+0x40/0x40 [45765.362981] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.370285] [] ? insert_kthread_work+0x40/0x40 [45765.377115] ib_nl_sa_wq S ffff99ec6b95a6e0 0 543 2 0x00000000 [45765.385059] Call Trace: [45765.387805] [] schedule+0x29/0x70 [45765.393379] [] rescuer_thread+0x31e/0x3c0 [45765.399718] [] ? __schedule+0x320/0x680 [45765.405851] [] ? process_one_work+0x440/0x440 [45765.412579] [] kthread+0xd1/0xe0 [45765.418057] [] ? insert_kthread_work+0x40/0x40 [45765.424882] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.432184] [] ? insert_kthread_work+0x40/0x40 [45765.439014] mlx4_ib S ffff99ec6a411660 0 545 2 0x00000000 [45765.446954] Call Trace: [45765.449699] [] schedule+0x29/0x70 [45765.455264] [] rescuer_thread+0x31e/0x3c0 [45765.461613] [] ? __schedule+0x320/0x680 [45765.467768] [] ? process_one_work+0x440/0x440 [45765.474493] [] kthread+0xd1/0xe0 [45765.479955] [] ? insert_kthread_work+0x40/0x40 [45765.486773] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.494083] [] ? insert_kthread_work+0x40/0x40 [45765.500921] mlx4_ib_mcg S ffff99ec6a4126e0 0 546 2 0x00000000 [45765.508879] Call Trace: [45765.511629] [] schedule+0x29/0x70 [45765.517194] [] rescuer_thread+0x31e/0x3c0 [45765.523533] [] ? __schedule+0x320/0x680 [45765.529681] [] ? process_one_work+0x440/0x440 [45765.536426] [] kthread+0xd1/0xe0 [45765.541891] [] ? insert_kthread_work+0x40/0x40 [45765.548711] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.556027] [] ? insert_kthread_work+0x40/0x40 [45765.562858] ib_mad1 S ffff99ec6a413760 0 547 2 0x00000000 [45765.570819] Call Trace: [45765.573575] [] schedule+0x29/0x70 [45765.579136] [] rescuer_thread+0x31e/0x3c0 [45765.585499] [] ? __schedule+0x320/0x680 [45765.591654] [] ? process_one_work+0x440/0x440 [45765.598390] [] kthread+0xd1/0xe0 [45765.603871] [] ? insert_kthread_work+0x40/0x40 [45765.610696] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.618007] [] ? insert_kthread_work+0x40/0x40 [45765.624838] to_fifo S ffff99ec6a4147e0 0 548 2 0x00000000 [45765.632787] Call Trace: [45765.635534] [] schedule+0x29/0x70 [45765.641100] [] rescuer_thread+0x31e/0x3c0 [45765.647456] [] ? __schedule+0x320/0x680 [45765.653594] [] ? process_one_work+0x440/0x440 [45765.660327] [] kthread+0xd1/0xe0 [45765.665797] [] ? insert_kthread_work+0x40/0x40 [45765.672614] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.679916] [] ? insert_kthread_work+0x40/0x40 [45765.686745] ib_mad2 S ffff99ec6a415860 0 549 2 0x00000000 [45765.694686] Call Trace: [45765.697441] [] schedule+0x29/0x70 [45765.703006] [] rescuer_thread+0x31e/0x3c0 [45765.709348] [] ? __schedule+0x320/0x680 [45765.715517] [] ? process_one_work+0x440/0x440 [45765.722238] [] kthread+0xd1/0xe0 [45765.727712] [] ? insert_kthread_work+0x40/0x40 [45765.734540] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.741849] [] ? insert_kthread_work+0x40/0x40 [45765.748681] to_fifo S ffff99ec6a4168e0 0 550 2 0x00000000 [45765.756632] Call Trace: [45765.759387] [] schedule+0x29/0x70 [45765.764958] [] rescuer_thread+0x31e/0x3c0 [45765.771290] [] ? __schedule+0x320/0x680 [45765.777450] [] ? process_one_work+0x440/0x440 [45765.784187] [] kthread+0xd1/0xe0 [45765.789663] [] ? insert_kthread_work+0x40/0x40 [45765.796494] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.803807] [] ? insert_kthread_work+0x40/0x40 [45765.810632] kworker/9:2 S ffff99ec6a4b0000 0 572 2 0x00000000 [45765.818564] Call Trace: [45765.821324] [] schedule+0x29/0x70 [45765.826881] [] worker_thread+0x1d9/0x3c0 [45765.833134] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45765.840462] [] kthread+0xd1/0xe0 [45765.845938] [] ? insert_kthread_work+0x40/0x40 [45765.852772] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.860090] [] ? insert_kthread_work+0x40/0x40 [45765.866901] kworker/0:1H S ffff99e84e7c4200 0 579 2 0x00000000 [45765.874845] Call Trace: [45765.877591] [] schedule+0x29/0x70 [45765.883147] [] worker_thread+0x1d9/0x3c0 [45765.889393] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45765.896702] [] kthread+0xd1/0xe0 [45765.902164] [] ? insert_kthread_work+0x40/0x40 [45765.909006] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.916325] [] ? insert_kthread_work+0x40/0x40 [45765.923148] jbd2/sda1-8 S ffff99e84e7c5280 0 589 2 0x00000000 [45765.931095] Call Trace: [45765.933848] [] schedule+0x29/0x70 [45765.939463] [] kjournald2+0x203/0x260 [jbd2] [45765.946100] [] ? wake_up_atomic_t+0x30/0x30 [45765.952642] [] ? commit_timeout+0x10/0x10 [jbd2] [45765.959662] [] kthread+0xd1/0xe0 [45765.965137] [] ? insert_kthread_work+0x40/0x40 [45765.971950] [] ret_from_fork_nospec_begin+0x21/0x21 [45765.979262] [] ? insert_kthread_work+0x40/0x40 [45765.986086] ext4-rsv-conver S ffff99e84dd885e0 0 590 2 0x00000000 [45765.994042] Call Trace: [45765.996796] [] schedule+0x29/0x70 [45766.002356] [] rescuer_thread+0x31e/0x3c0 [45766.008697] [] ? __schedule+0x320/0x680 [45766.014842] [] ? process_one_work+0x440/0x440 [45766.021571] [] kthread+0xd1/0xe0 [45766.027033] [] ? insert_kthread_work+0x40/0x40 [45766.033859] [] ret_from_fork_nospec_begin+0x21/0x21 [45766.041171] [] ? insert_kthread_work+0x40/0x40 [45766.048067] systemd-journal S ffff99e8f5595280 0 672 1 0x00000000 [45766.056019] Call Trace: [45766.058766] [] schedule+0x29/0x70 [45766.064331] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45766.072214] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [45766.079816] [] schedule_hrtimeout_range+0x13/0x20 [45766.086925] [] ep_poll+0x23e/0x360 [45766.092581] [] ? do_truncate+0x88/0xc0 [45766.098624] [] ? __sb_start_write+0x58/0x120 [45766.105264] [] ? wake_up_state+0x20/0x20 [45766.111530] [] SyS_epoll_wait+0xed/0x120 [45766.117773] [] ? system_call_after_swapgs+0x96/0x13a [45766.125183] [] system_call_fastpath+0x25/0x2a [45766.131900] [] ? system_call_after_swapgs+0xa2/0x13a [45766.139300] rpciod S ffff99ec6df09660 0 689 2 0x00000000 [45766.147278] Call Trace: [45766.150038] [] schedule+0x29/0x70 [45766.155604] [] rescuer_thread+0x31e/0x3c0 [45766.161931] [] ? __schedule+0x320/0x680 [45766.168092] [] ? process_one_work+0x440/0x440 [45766.174827] [] kthread+0xd1/0xe0 [45766.180295] [] ? insert_kthread_work+0x40/0x40 [45766.187112] [] ret_from_fork_nospec_begin+0x21/0x21 [45766.194416] [] ? insert_kthread_work+0x40/0x40 [45766.201270] xprtiod S ffff99ec6df0a6e0 0 690 2 0x00000000 [45766.209250] Call Trace: [45766.211997] [] schedule+0x29/0x70 [45766.217562] [] rescuer_thread+0x31e/0x3c0 [45766.223903] [] ? __schedule+0x320/0x680 [45766.230057] [] ? process_one_work+0x440/0x440 [45766.236779] [] kthread+0xd1/0xe0 [45766.242247] [] ? insert_kthread_work+0x40/0x40 [45766.249074] [] ret_from_fork_nospec_begin+0x21/0x21 [45766.256379] [] ? insert_kthread_work+0x40/0x40 [45766.263217] lvmetad S ffff99ec6bab68e0 0 702 1 0x00000000 [45766.271153] Call Trace: [45766.273906] [] schedule+0x29/0x70 [45766.279471] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45766.287363] [] ? add_wait_queue+0x3c/0x50 [45766.293713] [] ? __pollwait+0x7f/0xf0 [45766.299673] [] ? list_del+0xd/0x30 [45766.305343] [] schedule_hrtimeout_range+0x13/0x20 [45766.312481] [] poll_schedule_timeout+0x55/0xc0 [45766.319297] [] do_select+0x6d1/0x7c0 [45766.321351] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [45766.336628] [] ? __pollwait+0xf0/0xf0 [45766.342583] [] ? zone_statistics+0x88/0xa0 [45766.349023] [] ? get_page_from_freelist+0x36f/0xac0 [45766.356341] [] ? list_del+0xd/0x30 [45766.361993] [] ? __rmqueue+0x8a/0x460 [45766.367954] [] ? zone_statistics+0x88/0xa0 [45766.374395] [] core_sys_select+0x1ff/0x340 [45766.380832] [] ? enqueue_entity+0x2ef/0xbe0 [45766.387361] [] ? cpumask_next_and+0x35/0x50 [45766.393894] [] ? enqueue_task_fair+0x208/0x6c0 [45766.400721] [] ? sched_clock_cpu+0x85/0xc0 [45766.407153] [] ? recalc_sigpending+0x1b/0x70 [45766.413777] [] ? __set_task_blocked+0x41/0xa0 [45766.420511] [] ? __set_current_blocked+0x42/0x70 [45766.427541] [] SyS_pselect6+0x153/0x260 [45766.433683] [] ? system_call_after_swapgs+0x96/0x13a [45766.441089] [] ? system_call_after_swapgs+0xa2/0x13a [45766.448481] [] ? system_call_after_swapgs+0x96/0x13a [45766.455882] [] ? system_call_after_swapgs+0xa2/0x13a [45766.463290] [] system_call_fastpath+0x25/0x2a [45766.470015] [] ? system_call_after_swapgs+0xa2/0x13a [45766.477428] kmpathd S ffff99ec6df0e8e0 0 716 2 0x00000000 [45766.485421] Call Trace: [45766.488174] [] schedule+0x29/0x70 [45766.493746] [] rescuer_thread+0x31e/0x3c0 [45766.500089] [] ? __schedule+0x320/0x680 [45766.506237] [] ? process_one_work+0x440/0x440 [45766.512964] [] kthread+0xd1/0xe0 [45766.518432] [] ? insert_kthread_work+0x40/0x40 [45766.525282] [] ret_from_fork_nospec_begin+0x21/0x21 [45766.532592] [] ? insert_kthread_work+0x40/0x40 [45766.539422] kmpath_handlerd S ffff99ec6df0d860 0 717 2 0x00000000 [45766.547387] Call Trace: [45766.550147] [] schedule+0x29/0x70 [45766.555720] [] rescuer_thread+0x31e/0x3c0 [45766.562053] [] ? __schedule+0x320/0x680 [45766.568191] [] ? process_one_work+0x440/0x440 [45766.574911] [] kthread+0xd1/0xe0 [45766.580384] [] ? insert_kthread_work+0x40/0x40 [45766.587192] [] ret_from_fork_nospec_begin+0x21/0x21 [45766.594497] [] ? insert_kthread_work+0x40/0x40 [45766.601334] multipathd S ffff99ec6a4b1080 0 725 1 0x00000000 [45766.609303] Call Trace: [45766.612055] [] schedule+0x29/0x70 [45766.617611] [] futex_wait_queue_me+0xc6/0x130 [45766.624346] [] futex_wait+0x17b/0x280 [45766.630325] [] do_futex+0x106/0x4d0 [45766.636109] [] ? mntput+0x24/0x40 [45766.641659] [] ? __fput+0x195/0x230 [45766.647422] [] SyS_futex+0x80/0x190 [45766.653182] [] ? system_call_after_swapgs+0x96/0x13a [45766.660576] [] ? system_call_after_swapgs+0xa2/0x13a [45766.667977] [] ? system_call_after_swapgs+0x96/0x13a [45766.675383] [] ? system_call_after_swapgs+0xa2/0x13a [45766.682783] [] ? system_call_after_swapgs+0x96/0x13a [45766.690192] [] ? system_call_after_swapgs+0xa2/0x13a [45766.697591] [] system_call_fastpath+0x25/0x2a [45766.704340] [] ? system_call_after_swapgs+0xa2/0x13a [45766.711740] multipathd S ffff99e86c16d280 0 728 1 0x00000000 [45766.719680] Call Trace: [45766.722427] [] schedule+0x29/0x70 [45766.728000] [] futex_wait_queue_me+0xc6/0x130 [45766.734728] [] futex_wait+0x17b/0x280 [45766.740689] [] do_futex+0x106/0x4d0 [45766.746469] [] ? __switch_to+0x151/0x580 [45766.752704] [] SyS_futex+0x80/0x190 [45766.758487] [] ? system_call_after_swapgs+0x96/0x13a [45766.765896] [] ? system_call_after_swapgs+0xa2/0x13a [45766.773303] [] ? system_call_after_swapgs+0x96/0x13a [45766.780708] [] ? system_call_after_swapgs+0xa2/0x13a [45766.788114] [] ? system_call_after_swapgs+0x96/0x13a [45766.795513] [] ? system_call_after_swapgs+0xa2/0x13a [45766.802916] [] system_call_fastpath+0x25/0x2a [45766.809658] [] ? system_call_after_swapgs+0xa2/0x13a [45766.817062] multipathd S ffff99e86c16e300 0 729 1 0x00000000 [45766.824996] Call Trace: [45766.827743] [] schedule+0x29/0x70 [45766.833314] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45766.841213] [] ? add_wait_queue+0x3c/0x50 [45766.847547] [] ? __pollwait+0x7f/0xf0 [45766.853502] [] schedule_hrtimeout_range+0x13/0x20 [45766.860617] [] poll_schedule_timeout+0x55/0xc0 [45766.867438] [] do_sys_poll+0x48d/0x590 [45766.873507] [] ? netlink_recvmsg+0x212/0x490 [45766.880146] [] ? plist_add+0x91/0xd0 [45766.886002] [] ? enqueue_pushable_task+0x6d/0x90 [45766.893018] [] ? tracing_is_on+0x15/0x30 [45766.899254] [] ? tracing_record_cmdline+0x1d/0x120 [45766.906490] [] ? __pollwait+0xf0/0xf0 [45766.912448] [] ? try_to_wake_up+0x192/0x390 [45766.919006] [] ? wake_up_q+0x5b/0x80 [45766.924863] [] ? futex_wake_op+0x3b5/0x640 [45766.931307] [] ? do_futex+0x17a/0x4d0 [45766.937267] [] ? _raw_spin_unlock_bh+0x1e/0x20 [45766.944102] [] ? release_sock+0x128/0x180 [45766.950445] [] ? system_call_after_swapgs+0xa2/0x13a [45766.957852] [] SyS_poll+0x74/0x110 [45766.963507] [] ? system_call_after_swapgs+0x96/0x13a [45766.970905] [] ? system_call_after_swapgs+0xa2/0x13a [45766.978313] [] system_call_fastpath+0x25/0x2a [45766.985044] [] ? system_call_after_swapgs+0xa2/0x13a [45766.992442] multipathd S ffff99e86c169080 0 730 1 0x00000000 [45767.000399] Call Trace: [45767.003152] [] schedule+0x29/0x70 [45767.008718] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45767.016511] [] ? hrtimer_get_res+0x50/0x50 [45767.022949] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45767.030939] [] schedule_hrtimeout_range+0x13/0x20 [45767.038061] [] poll_schedule_timeout+0x55/0xc0 [45767.044889] [] do_sys_poll+0x48d/0x590 [45767.050999] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [45767.058597] [] ? add_page_to_unevictable_list+0xc8/0x100 [45767.066393] [] ? __pollwait+0xf0/0xf0 [45767.072354] [] ? unlock_page+0x2b/0x30 [45767.078404] [] ? follow_page_mask+0x6ef/0x9f0 [45767.085131] [] ? __get_user_pages+0x16c/0x7e0 [45767.091844] [] ? change_protection+0x3a9/0x680 [45767.098663] [] ? __kmalloc+0x55/0x230 [45767.104616] [] ? __check_object_size+0x198/0x250 [45767.111635] [] ? recalc_sigpending+0x1b/0x70 [45767.118251] [] ? __set_task_blocked+0x41/0xa0 [45767.124985] [] SyS_ppoll+0xee/0x1f0 [45767.130737] [] system_call_fastpath+0x25/0x2a [45767.137465] [] ? system_call_after_swapgs+0xa2/0x13a [45767.144895] multipathd S ffff99e86c16b180 0 732 1 0x00000000 [45767.152849] Call Trace: [45767.155597] [] schedule+0x29/0x70 [45767.161152] [] do_nanosleep+0x70/0x100 [45767.167203] [] hrtimer_nanosleep+0xbb/0x180 [45767.173736] [] ? hrtimer_get_res+0x50/0x50 [45767.180173] [] ? do_nanosleep+0x5b/0x100 [45767.186417] [] SyS_nanosleep+0x96/0xb0 [45767.192462] [] system_call_fastpath+0x25/0x2a [45767.199216] [] ? system_call_after_swapgs+0xa2/0x13a [45767.206612] multipathd S ffff99e86c16a100 0 733 1 0x00000000 [45767.214568] Call Trace: [45767.217328] [] schedule+0x29/0x70 [45767.222886] [] futex_wait_queue_me+0xc6/0x130 [45767.229615] [] futex_wait+0x17b/0x280 [45767.235574] [] ? mntput+0x24/0x40 [45767.241146] [] ? path_openat+0x176/0x5a0 [45767.247389] [] do_futex+0x106/0x4d0 [45767.253140] [] ? mntput+0x24/0x40 [45767.258703] [] ? __fput+0x195/0x230 [45767.264460] [] SyS_futex+0x80/0x190 [45767.270234] [] ? system_call_after_swapgs+0x96/0x13a [45767.277641] [] ? system_call_after_swapgs+0xa2/0x13a [45767.285044] [] ? system_call_after_swapgs+0x96/0x13a [45767.292436] [] ? system_call_after_swapgs+0xa2/0x13a [45767.299844] [] ? system_call_after_swapgs+0x96/0x13a [45767.307246] [] ? system_call_after_swapgs+0xa2/0x13a [45767.314653] [] system_call_fastpath+0x25/0x2a [45767.321375] [] ? system_call_after_swapgs+0xa2/0x13a [45767.328779] multipathd S ffff99ec6bab4200 0 866 1 0x00000000 [45767.336707] Call Trace: [45767.339469] [] schedule+0x29/0x70 [45767.345079] [] dm_wait_event+0xa1/0xf0 [dm_mod] [45767.351988] [] ? strlcpy+0x42/0x60 [45767.357650] [] ? wake_up_atomic_t+0x30/0x30 [45767.364199] [] dev_wait+0x50/0xd0 [dm_mod] [45767.370638] [] ctl_ioctl+0x24b/0x640 [dm_mod] [45767.377373] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [45767.384406] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [45767.391120] [] do_vfs_ioctl+0x3a0/0x5b0 [45767.397254] [] ? __set_task_blocked+0x41/0xa0 [45767.403976] [] ? __set_current_blocked+0x42/0x70 [45767.410988] [] SyS_ioctl+0xa1/0xc0 [45767.416651] [] ? system_call_after_swapgs+0x96/0x13a [45767.424060] [] system_call_fastpath+0x25/0x2a [45767.430790] [] ? system_call_after_swapgs+0xa2/0x13a [45767.438208] multipathd S ffff99ec6bab26e0 0 873 1 0x00000000 [45767.446175] Call Trace: [45767.448934] [] schedule+0x29/0x70 [45767.454538] [] dm_wait_event+0xa1/0xf0 [dm_mod] [45767.461461] [] ? strlcpy+0x42/0x60 [45767.467124] [] ? wake_up_atomic_t+0x30/0x30 [45767.473664] [] dev_wait+0x50/0xd0 [dm_mod] [45767.480107] [] ctl_ioctl+0x24b/0x640 [dm_mod] [45767.486822] [] ? __enqueue_entity+0x78/0x80 [45767.493354] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [45767.500380] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [45767.507108] [] do_vfs_ioctl+0x3a0/0x5b0 [45767.513256] [] ? __set_task_blocked+0x41/0xa0 [45767.519984] [] ? __set_current_blocked+0x42/0x70 [45767.526995] [] SyS_ioctl+0xa1/0xc0 [45767.532657] [] ? system_call_after_swapgs+0x96/0x13a [45767.540058] [] system_call_fastpath+0x25/0x2a [45767.546778] [] ? system_call_after_swapgs+0xa2/0x13a [45767.554197] multipathd S ffff99ec6b45f6c0 0 877 1 0x00000000 [45767.562140] Call Trace: [45767.564892] [] schedule+0x29/0x70 [45767.570466] [] dm_wait_event+0xa1/0xf0 [dm_mod] [45767.577397] [] ? strlcpy+0x42/0x60 [45767.583066] [] ? wake_up_atomic_t+0x30/0x30 [45767.589609] [] dev_wait+0x50/0xd0 [dm_mod] [45767.596053] [] ctl_ioctl+0x24b/0x640 [dm_mod] [45767.602775] [] ? __enqueue_entity+0x78/0x80 [45767.609314] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [45767.616342] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [45767.623070] [] do_vfs_ioctl+0x3a0/0x5b0 [45767.629215] [] ? __set_task_blocked+0x41/0xa0 [45767.635944] [] ? __set_current_blocked+0x42/0x70 [45767.642962] [] SyS_ioctl+0xa1/0xc0 [45767.648626] [] ? system_call_after_swapgs+0x96/0x13a [45767.656040] [] system_call_fastpath+0x25/0x2a [45767.662770] [] ? system_call_after_swapgs+0xa2/0x13a [45767.670167] multipathd S ffff99e86d0b0000 0 883 1 0x00000000 [45767.678113] Call Trace: [45767.680859] [] schedule+0x29/0x70 [45767.686436] [] dm_wait_event+0xa1/0xf0 [dm_mod] [45767.693357] [] ? strlcpy+0x42/0x60 [45767.699008] [] ? wake_up_atomic_t+0x30/0x30 [45767.705584] [] dev_wait+0x50/0xd0 [dm_mod] [45767.712022] [] ctl_ioctl+0x24b/0x640 [dm_mod] [45767.718757] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [45767.725782] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [45767.732534] [] do_vfs_ioctl+0x3a0/0x5b0 [45767.738674] [] ? __set_task_blocked+0x41/0xa0 [45767.745404] [] ? __set_current_blocked+0x42/0x70 [45767.752423] [] SyS_ioctl+0xa1/0xc0 [45767.758085] [] ? system_call_after_swapgs+0x96/0x13a [45767.765523] [] system_call_fastpath+0x25/0x2a [45767.772251] [] ? system_call_after_swapgs+0xa2/0x13a [45767.779662] multipathd S ffff99ec6ba3a6e0 0 887 1 0x00000000 [45767.787624] Call Trace: [45767.790376] [] schedule+0x29/0x70 [45767.795947] [] dm_wait_event+0xa1/0xf0 [dm_mod] [45767.802864] [] ? strlcpy+0x42/0x60 [45767.808534] [] ? wake_up_atomic_t+0x30/0x30 [45767.815081] [] dev_wait+0x50/0xd0 [dm_mod] [45767.821544] [] ctl_ioctl+0x24b/0x640 [dm_mod] [45767.828296] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [45767.835334] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [45767.842074] [] do_vfs_ioctl+0x3a0/0x5b0 [45767.848220] [] ? __set_task_blocked+0x41/0xa0 [45767.854951] [] ? __set_current_blocked+0x42/0x70 [45767.861955] [] SyS_ioctl+0xa1/0xc0 [45767.867611] [] ? system_call_after_swapgs+0x96/0x13a [45767.875012] [] system_call_fastpath+0x25/0x2a [45767.881732] [] ? system_call_after_swapgs+0xa2/0x13a [45767.889135] systemd-udevd S ffff99ec6bab1080 0 727 1 0x00000000 [45767.897084] Call Trace: [45767.899837] [] schedule+0x29/0x70 [45767.905404] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45767.913197] [] ? hrtimer_get_res+0x50/0x50 [45767.919636] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45767.927635] [] schedule_hrtimeout_range+0x13/0x20 [45767.934757] [] ep_poll+0x23e/0x360 [45767.940425] [] ? do_unlinkat+0xf1/0x2d0 [45767.946563] [] ? wake_up_state+0x20/0x20 [45767.952826] [] SyS_epoll_wait+0xed/0x120 [45767.959078] [] ? system_call_after_swapgs+0x96/0x13a [45767.966496] [] system_call_fastpath+0x25/0x2a [45767.973239] [] ? system_call_after_swapgs+0xa2/0x13a [45767.980646] kipmi0 S ffff99e86c015280 0 806 2 0x00000000 [45767.988597] Call Trace: [45767.991351] [] schedule+0x29/0x70 [45767.996916] [] ipmi_thread+0x23d/0x250 [ipmi_si] [45768.003942] [] ? cleanup_one_si+0x60/0x60 [ipmi_si] [45768.011247] [] kthread+0xd1/0xe0 [45768.016702] [] ? insert_kthread_work+0x40/0x40 [45768.023544] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.030862] [] ? insert_kthread_work+0x40/0x40 [45768.037685] kworker/11:1H S ffff99ec6a7f0000 0 813 2 0x00000000 [45768.045720] Call Trace: [45768.048471] [] schedule+0x29/0x70 [45768.054038] [] worker_thread+0x1d9/0x3c0 [45768.060283] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45768.067604] [] kthread+0xd1/0xe0 [45768.073087] [] ? insert_kthread_work+0x40/0x40 [45768.079913] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.087247] [] ? insert_kthread_work+0x40/0x40 [45768.094090] kworker/14:1H S ffff99ec6a7f1080 0 814 2 0x00000000 [45768.102037] Call Trace: [45768.104797] [] schedule+0x29/0x70 [45768.110364] [] worker_thread+0x1d9/0x3c0 [45768.116604] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45768.123914] [] kthread+0xd1/0xe0 [45768.129381] [] ? insert_kthread_work+0x40/0x40 [45768.136209] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.143541] [] ? insert_kthread_work+0x40/0x40 [45768.150374] kworker/1:1H S ffff99e6b2009080 0 836 2 0x00000000 [45768.158343] Call Trace: [45768.161098] [] schedule+0x29/0x70 [45768.166648] [] worker_thread+0x1d9/0x3c0 [45768.172892] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45768.180210] [] kthread+0xd1/0xe0 [45768.185670] [] ? insert_kthread_work+0x40/0x40 [45768.192491] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.199795] [] ? insert_kthread_work+0x40/0x40 [45768.206621] kworker/25:1H S ffff99ec6a7f6300 0 859 2 0x00000000 [45768.214625] Call Trace: [45768.217395] [] schedule+0x29/0x70 [45768.222967] [] worker_thread+0x1d9/0x3c0 [45768.229203] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45768.236535] [] kthread+0xd1/0xe0 [45768.241999] [] ? insert_kthread_work+0x40/0x40 [45768.248826] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.256124] [] ? insert_kthread_work+0x40/0x40 [45768.262953] kdmflush S ffff99ec6a7f5860 0 860 2 0x00000000 [45768.270902] Call Trace: [45768.273668] [] schedule+0x29/0x70 [45768.279247] [] rescuer_thread+0x31e/0x3c0 [45768.285578] [] ? __schedule+0x320/0x680 [45768.291723] [] ? process_one_work+0x440/0x440 [45768.298444] [] kthread+0xd1/0xe0 [45768.303905] [] ? insert_kthread_work+0x40/0x40 [45768.310716] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.318027] [] ? insert_kthread_work+0x40/0x40 [45768.324855] bioset S ffff99ec6a7f47e0 0 864 2 0x00000000 [45768.332820] Call Trace: [45768.335572] [] schedule+0x29/0x70 [45768.341144] [] rescuer_thread+0x31e/0x3c0 [45768.347488] [] ? __schedule+0x320/0x680 [45768.353625] [] ? process_one_work+0x440/0x440 [45768.360347] [] kthread+0xd1/0xe0 [45768.365815] [] ? insert_kthread_work+0x40/0x40 [45768.372645] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.379953] [] ? insert_kthread_work+0x40/0x40 [45768.386776] kdmwork-253:0 S ffff99ec6a7f3180 0 865 2 0x00000000 [45768.394703] Call Trace: [45768.397464] [] schedule+0x29/0x70 [45768.403031] [] kthread_worker_fn+0xf7/0x180 [45768.409569] [] ? kthread_stop+0xf0/0xf0 [45768.415713] [] kthread+0xd1/0xe0 [45768.421189] [] ? insert_kthread_work+0x40/0x40 [45768.428007] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.435310] [] ? insert_kthread_work+0x40/0x40 [45768.442145] kdmflush S ffff99ec6a7f26e0 0 870 2 0x00000000 [45768.450115] Call Trace: [45768.452868] [] schedule+0x29/0x70 [45768.458436] [] rescuer_thread+0x31e/0x3c0 [45768.464777] [] ? __schedule+0x320/0x680 [45768.470935] [] ? process_one_work+0x440/0x440 [45768.477671] [] kthread+0xd1/0xe0 [45768.483130] [] ? insert_kthread_work+0x40/0x40 [45768.489949] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.497261] [] ? insert_kthread_work+0x40/0x40 [45768.504091] bioset S ffff99e6b200c7e0 0 871 2 0x00000000 [45768.512039] Call Trace: [45768.514777] [] schedule+0x29/0x70 [45768.520347] [] rescuer_thread+0x31e/0x3c0 [45768.526673] [] ? __schedule+0x320/0x680 [45768.532820] [] ? process_one_work+0x440/0x440 [45768.539570] [] kthread+0xd1/0xe0 [45768.545030] [] ? insert_kthread_work+0x40/0x40 [45768.551842] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.559159] [] ? insert_kthread_work+0x40/0x40 [45768.565990] kdmwork-253:1 S ffff99e6b200e300 0 872 2 0x00000000 [45768.573935] Call Trace: [45768.576675] [] schedule+0x29/0x70 [45768.582241] [] kthread_worker_fn+0xf7/0x180 [45768.588778] [] ? kthread_stop+0xf0/0xf0 [45768.594925] [] kthread+0xd1/0xe0 [45768.600399] [] ? insert_kthread_work+0x40/0x40 [45768.607231] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.614557] [] ? insert_kthread_work+0x40/0x40 [45768.621378] kdmflush S ffff99e6b200b760 0 874 2 0x00000000 [45768.629298] Call Trace: [45768.632043] [] schedule+0x29/0x70 [45768.637622] [] rescuer_thread+0x31e/0x3c0 [45768.643962] [] ? __schedule+0x320/0x680 [45768.650117] [] ? process_one_work+0x440/0x440 [45768.656830] [] kthread+0xd1/0xe0 [45768.662313] [] ? insert_kthread_work+0x40/0x40 [45768.669145] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.676446] [] ? insert_kthread_work+0x40/0x40 [45768.683268] bioset S ffff99e6b200d860 0 875 2 0x00000000 [45768.691210] Call Trace: [45768.693961] [] schedule+0x29/0x70 [45768.699520] [] rescuer_thread+0x31e/0x3c0 [45768.705863] [] ? __schedule+0x320/0x680 [45768.712025] [] ? process_one_work+0x440/0x440 [45768.718757] [] kthread+0xd1/0xe0 [45768.724232] [] ? insert_kthread_work+0x40/0x40 [45768.731064] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.738374] [] ? insert_kthread_work+0x40/0x40 [45768.745197] kdmwork-253:2 S ffff99e6b200a6e0 0 876 2 0x00000000 [45768.753157] Call Trace: [45768.755903] [] schedule+0x29/0x70 [45768.761475] [] kthread_worker_fn+0xf7/0x180 [45768.768008] [] ? kthread_stop+0xf0/0xf0 [45768.774153] [] kthread+0xd1/0xe0 [45768.779628] [] ? insert_kthread_work+0x40/0x40 [45768.786463] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.793780] [] ? insert_kthread_work+0x40/0x40 [45768.800595] kdmflush S ffff99e867ec9660 0 880 2 0x00000000 [45768.808564] Call Trace: [45768.811318] [] schedule+0x29/0x70 [45768.816867] [] rescuer_thread+0x31e/0x3c0 [45768.823209] [] ? __schedule+0x320/0x680 [45768.829356] [] ? process_one_work+0x440/0x440 [45768.836083] [] kthread+0xd1/0xe0 [45768.841571] [] ? insert_kthread_work+0x40/0x40 [45768.848398] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.855695] [] ? insert_kthread_work+0x40/0x40 [45768.862527] bioset S ffff99e867eca6e0 0 881 2 0x00000000 [45768.870477] Call Trace: [45768.873229] [] schedule+0x29/0x70 [45768.878795] [] rescuer_thread+0x31e/0x3c0 [45768.885143] [] ? __schedule+0x320/0x680 [45768.891289] [] ? process_one_work+0x440/0x440 [45768.898016] [] kthread+0xd1/0xe0 [45768.903489] [] ? insert_kthread_work+0x40/0x40 [45768.910314] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.917648] [] ? insert_kthread_work+0x40/0x40 [45768.924475] kdmwork-253:3 S ffff99e867ecb180 0 882 2 0x00000000 [45768.932451] Call Trace: [45768.935196] [] schedule+0x29/0x70 [45768.940762] [] kthread_worker_fn+0xf7/0x180 [45768.947289] [] ? kthread_stop+0xf0/0xf0 [45768.953434] [] kthread+0xd1/0xe0 [45768.958903] [] ? insert_kthread_work+0x40/0x40 [45768.965730] [] ret_from_fork_nospec_begin+0x21/0x21 [45768.973040] [] ? insert_kthread_work+0x40/0x40 [45768.979872] kdmflush S ffff99e867ecc7e0 0 884 2 0x00000000 [45768.987841] Call Trace: [45768.990601] [] schedule+0x29/0x70 [45768.996167] [] rescuer_thread+0x31e/0x3c0 [45769.002511] [] ? __schedule+0x320/0x680 [45769.008649] [] ? process_one_work+0x440/0x440 [45769.015379] [] kthread+0xd1/0xe0 [45769.020846] [] ? insert_kthread_work+0x40/0x40 [45769.027665] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.034974] [] ? insert_kthread_work+0x40/0x40 [45769.041782] bioset S ffff99e867ecd860 0 885 2 0x00000000 [45769.049716] Call Trace: [45769.052476] [] schedule+0x29/0x70 [45769.058043] [] rescuer_thread+0x31e/0x3c0 [45769.064383] [] ? __schedule+0x320/0x680 [45769.070523] [] ? process_one_work+0x440/0x440 [45769.077250] [] kthread+0xd1/0xe0 [45769.082704] [] ? insert_kthread_work+0x40/0x40 [45769.089532] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.096835] [] ? insert_kthread_work+0x40/0x40 [45769.103660] kdmwork-253:4 S ffff99e867ece8e0 0 886 2 0x00000000 [45769.111599] Call Trace: [45769.114358] [] schedule+0x29/0x70 [45769.119916] [] kthread_worker_fn+0xf7/0x180 [45769.126437] [] ? kthread_stop+0xf0/0xf0 [45769.132598] [] kthread+0xd1/0xe0 [45769.138064] [] ? insert_kthread_work+0x40/0x40 [45769.144890] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.152202] [] ? insert_kthread_work+0x40/0x40 [45769.159032] kworker/9:1H S ffff99ec697e8000 0 893 2 0x00000000 [45769.166972] Call Trace: [45769.169724] [] schedule+0x29/0x70 [45769.175305] [] worker_thread+0x1d9/0x3c0 [45769.181536] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.188840] [] kthread+0xd1/0xe0 [45769.194309] [] ? insert_kthread_work+0x40/0x40 [45769.201128] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.208455] [] ? insert_kthread_work+0x40/0x40 [45769.215285] kworker/10:1H S ffff99ec697e9080 0 894 2 0x00000000 [45769.223232] Call Trace: [45769.225979] [] schedule+0x29/0x70 [45769.231531] [] worker_thread+0x1d9/0x3c0 [45769.237785] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.245103] [] kthread+0xd1/0xe0 [45769.250597] [] ? insert_kthread_work+0x40/0x40 [45769.257415] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.264726] [] ? insert_kthread_work+0x40/0x40 [45769.271544] kworker/12:1H S ffff99ec697ea100 0 898 2 0x00000000 [45769.279502] Call Trace: [45769.282267] [] schedule+0x29/0x70 [45769.287830] [] worker_thread+0x1d9/0x3c0 [45769.294074] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.301382] [] kthread+0xd1/0xe0 [45769.306850] [] ? insert_kthread_work+0x40/0x40 [45769.313676] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.320985] [] ? insert_kthread_work+0x40/0x40 [45769.327816] kworker/31:1H S ffff99ec6d1e0000 0 904 2 0x00000000 [45769.335765] Call Trace: [45769.338516] [] schedule+0x29/0x70 [45769.344083] [] worker_thread+0x1d9/0x3c0 [45769.350328] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.357648] [] kthread+0xd1/0xe0 [45769.363117] [] ? insert_kthread_work+0x40/0x40 [45769.369951] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.377260] [] ? insert_kthread_work+0x40/0x40 [45769.384069] kworker/30:1H S ffff99ec6d1e1080 0 905 2 0x00000000 [45769.392029] Call Trace: [45769.394782] [] schedule+0x29/0x70 [45769.400347] [] worker_thread+0x1d9/0x3c0 [45769.406595] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.413898] [] kthread+0xd1/0xe0 [45769.419360] [] ? insert_kthread_work+0x40/0x40 [45769.426184] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.433502] [] ? insert_kthread_work+0x40/0x40 [45769.440339] kworker/28:1H S ffff99ec6d1e2100 0 906 2 0x00000000 [45769.448324] Call Trace: [45769.451084] [] schedule+0x29/0x70 [45769.456662] [] worker_thread+0x1d9/0x3c0 [45769.462898] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.470207] [] kthread+0xd1/0xe0 [45769.475672] [] ? insert_kthread_work+0x40/0x40 [45769.482490] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.489815] [] ? insert_kthread_work+0x40/0x40 [45769.496615] kworker/20:1H S ffff99e86a412100 0 910 2 0x00000000 [45769.504597] Call Trace: [45769.507352] [] schedule+0x29/0x70 [45769.512917] [] worker_thread+0x1d9/0x3c0 [45769.519168] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.526469] [] kthread+0xd1/0xe0 [45769.531939] [] ? insert_kthread_work+0x40/0x40 [45769.538757] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.546060] [] ? insert_kthread_work+0x40/0x40 [45769.552892] kworker/26:1H S ffff99ec6d820000 0 911 2 0x00000000 [45769.560879] Call Trace: [45769.563639] [] schedule+0x29/0x70 [45769.569219] [] worker_thread+0x1d9/0x3c0 [45769.575462] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.582765] [] kthread+0xd1/0xe0 [45769.588238] [] ? insert_kthread_work+0x40/0x40 [45769.595062] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.602369] [] ? insert_kthread_work+0x40/0x40 [45769.609200] kworker/2:1H S ffff99e86c3e6300 0 912 2 0x00000000 [45769.617161] Call Trace: [45769.619905] [] schedule+0x29/0x70 [45769.625484] [] worker_thread+0x1d9/0x3c0 [45769.631729] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.639039] [] kthread+0xd1/0xe0 [45769.644508] [] ? insert_kthread_work+0x40/0x40 [45769.651325] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.658628] [] ? insert_kthread_work+0x40/0x40 [45769.665457] kworker/15:1H S ffff99ec6d826300 0 932 2 0x00000000 [45769.673425] Call Trace: [45769.676170] [] schedule+0x29/0x70 [45769.681729] [] worker_thread+0x1d9/0x3c0 [45769.687982] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.695299] [] kthread+0xd1/0xe0 [45769.700761] [] ? insert_kthread_work+0x40/0x40 [45769.707604] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.714912] [] ? insert_kthread_work+0x40/0x40 [45769.721720] kworker/8:1H S ffff99ec6bfb9080 0 948 2 0x00000000 [45769.729663] Call Trace: [45769.732417] [] schedule+0x29/0x70 [45769.737982] [] worker_thread+0x1d9/0x3c0 [45769.744227] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.751525] [] kthread+0xd1/0xe0 [45769.757003] [] ? insert_kthread_work+0x40/0x40 [45769.763821] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.771126] [] ? insert_kthread_work+0x40/0x40 [45769.777953] kworker/13:1H S ffff99ec6bfb8000 0 952 2 0x00000000 [45769.785932] Call Trace: [45769.788687] [] schedule+0x29/0x70 [45769.794254] [] worker_thread+0x1d9/0x3c0 [45769.800497] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.807807] [] kthread+0xd1/0xe0 [45769.813282] [] ? insert_kthread_work+0x40/0x40 [45769.820101] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.827412] [] ? insert_kthread_work+0x40/0x40 [45769.834256] kworker/27:1H S ffff99ec6a530000 0 1036 2 0x00000000 [45769.842243] Call Trace: [45769.844985] [] schedule+0x29/0x70 [45769.850554] [] worker_thread+0x1d9/0x3c0 [45769.856791] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.864093] [] kthread+0xd1/0xe0 [45769.869566] [] ? insert_kthread_work+0x40/0x40 [45769.876378] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.883688] [] ? insert_kthread_work+0x40/0x40 [45769.890512] kworker/2:3 S ffff99e86b14d280 0 1040 2 0x00000000 [45769.898480] Call Trace: [45769.901226] [] schedule+0x29/0x70 [45769.906783] [] worker_thread+0x1d9/0x3c0 [45769.913027] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45769.920338] [] kthread+0xd1/0xe0 [45769.925804] [] ? insert_kthread_work+0x40/0x40 [45769.932648] [] ret_from_fork_nospec_begin+0x21/0x21 [45769.939979] [] ? insert_kthread_work+0x40/0x40 [45769.946804] auditd S ffff99ec6b5a4200 0 1067 1 0x00000000 [45769.954754] Call Trace: [45769.957498] [] schedule+0x29/0x70 [45769.963063] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45769.970853] [] ? hrtimer_get_res+0x50/0x50 [45769.977284] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45769.985280] [] schedule_hrtimeout_range+0x13/0x20 [45769.992397] [] ep_poll+0x23e/0x360 [45769.998060] [] ? wake_up_state+0x20/0x20 [45770.004311] [] SyS_epoll_wait+0xed/0x120 [45770.010564] [] ? system_call_after_swapgs+0x96/0x13a [45770.017974] [] system_call_fastpath+0x25/0x2a [45770.024708] [] ? system_call_after_swapgs+0xa2/0x13a [45770.032113] auditd S ffff99e86a4e5280 0 1068 1 0x00000000 [45770.040064] Call Trace: [45770.042800] [] schedule+0x29/0x70 [45770.048375] [] futex_wait_queue_me+0xc6/0x130 [45770.055102] [] futex_wait+0x17b/0x280 [45770.061053] [] ? put_prev_entity+0x31/0x400 [45770.067588] [] ? __switch_to+0x151/0x580 [45770.073851] [] do_futex+0x106/0x4d0 [45770.079648] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [45770.087740] [] SyS_futex+0x80/0x190 [45770.093487] [] ? system_call_after_swapgs+0x96/0x13a [45770.100889] [] ? system_call_after_swapgs+0xa2/0x13a [45770.108295] [] ? system_call_after_swapgs+0x96/0x13a [45770.115698] [] ? system_call_after_swapgs+0xa2/0x13a [45770.123098] [] ? system_call_after_swapgs+0x96/0x13a [45770.130519] [] ? system_call_after_swapgs+0xa2/0x13a [45770.137928] [] system_call_fastpath+0x25/0x2a [45770.144649] [] ? system_call_after_swapgs+0xa2/0x13a [45770.152048] dbus-daemon S ffff99e8f5593180 0 1093 1 0x00000080 [45770.160025] Call Trace: [45770.162773] [] schedule+0x29/0x70 [45770.168346] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45770.176239] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [45770.183840] [] schedule_hrtimeout_range+0x13/0x20 [45770.190955] [] ep_poll+0x23e/0x360 [45770.196640] [] ? wake_up_state+0x20/0x20 [45770.202885] [] SyS_epoll_wait+0xed/0x120 [45770.209122] [] ? system_call_after_swapgs+0x96/0x13a [45770.216531] [] system_call_fastpath+0x25/0x2a [45770.223259] [] ? system_call_after_swapgs+0xa2/0x13a [45770.230668] rpcbind S ffff99ec6bbd4200 0 1096 1 0x00000080 [45770.238641] Call Trace: [45770.241387] [] schedule+0x29/0x70 [45770.246944] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45770.254739] [] ? hrtimer_get_res+0x50/0x50 [45770.261191] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45770.269188] [] schedule_hrtimeout_range+0x13/0x20 [45770.276304] [] poll_schedule_timeout+0x55/0xc0 [45770.283132] [] do_sys_poll+0x48d/0x590 [45770.289200] [] ? task_rq_unlock+0x20/0x20 [45770.295549] [] ? __wake_up_sync_key+0x1d/0x20 [45770.302269] [] ? __pollwait+0xf0/0xf0 [45770.308223] [] ? __pollwait+0xf0/0xf0 [45770.314170] [] ? __pollwait+0xf0/0xf0 [45770.320131] [] ? __pollwait+0xf0/0xf0 [45770.326084] [] ? __pollwait+0xf0/0xf0 [45770.332032] [] ? __pollwait+0xf0/0xf0 [45770.337991] [] ? __pollwait+0xf0/0xf0 [45770.343943] [] ? recalibrate_cpu_khz+0x10/0x10 [45770.350768] [] ? ktime_get_ts64+0x4a/0xf0 [45770.357107] [] ? ktime_get_ts64+0x52/0xf0 [45770.363447] [] SyS_poll+0x74/0x110 [45770.369111] [] system_call_fastpath+0x25/0x2a [45770.375865] [] ? system_call_after_swapgs+0xa2/0x13a [45770.383274] NetworkManager S ffff99ec6b431080 0 1100 1 0x00000080 [45770.391238] Call Trace: [45770.393985] [] schedule+0x29/0x70 [45770.399578] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45770.407381] [] ? hrtimer_get_res+0x50/0x50 [45770.413821] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45770.421811] [] schedule_hrtimeout_range+0x13/0x20 [45770.428929] [] poll_schedule_timeout+0x55/0xc0 [45770.435746] [] do_sys_poll+0x48d/0x590 [45770.441797] [] ? put_prev_entity+0x31/0x400 [45770.448331] [] ? __switch_to+0x151/0x580 [45770.454584] [] ? __schedule+0x320/0x680 [45770.460737] [] ? __pollwait+0xf0/0xf0 [45770.466709] [] ? __pollwait+0xf0/0xf0 [45770.472664] [] ? __pollwait+0xf0/0xf0 [45770.478637] [] ? __pollwait+0xf0/0xf0 [45770.484593] [] ? __pollwait+0xf0/0xf0 [45770.490544] [] ? __pollwait+0xf0/0xf0 [45770.496489] [] ? __pollwait+0xf0/0xf0 [45770.502442] [] ? __pollwait+0xf0/0xf0 [45770.508395] [] ? ktime_get_ts64+0x52/0xf0 [45770.514735] [] SyS_poll+0x74/0x110 [45770.520406] [] system_call_fastpath+0x25/0x2a [45770.527144] [] ? system_call_after_swapgs+0xa2/0x13a [45770.534562] gmain S ffff99ec6bbd0000 0 1238 1 0x00000080 [45770.542550] Call Trace: [45770.545317] [] schedule+0x29/0x70 [45770.550881] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45770.558675] [] ? hrtimer_get_res+0x50/0x50 [45770.565112] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45770.573102] [] schedule_hrtimeout_range+0x13/0x20 [45770.580231] [] poll_schedule_timeout+0x55/0xc0 [45770.587056] [] do_sys_poll+0x48d/0x590 [45770.593117] [] ? security_inode_permission+0x22/0x30 [45770.600525] [] ? __inode_permission+0x52/0xd0 [45770.607252] [] ? __pollwait+0xf0/0xf0 [45770.613210] [] ? __pollwait+0xf0/0xf0 [45770.619162] [] ? path_lookupat+0x1ea/0x8d0 [45770.625605] [] ? kmem_cache_alloc+0x35/0x1f0 [45770.632242] [] ? getname_flags+0x4f/0x1a0 [45770.638587] [] ? putname+0x3d/0x60 [45770.644254] [] ? user_path_at_empty+0x72/0xc0 [45770.650982] [] ? ktime_get_ts64+0x52/0xf0 [45770.657313] [] SyS_poll+0x74/0x110 [45770.662960] [] system_call_fastpath+0x25/0x2a [45770.669680] [] ? system_call_after_swapgs+0xa2/0x13a [45770.677085] gdbus S ffff99ec6bbd1080 0 1245 1 0x00000080 [45770.685017] Call Trace: [45770.687757] [] schedule+0x29/0x70 [45770.693329] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45770.701230] [] ? add_wait_queue+0x3c/0x50 [45770.707569] [] ? __pollwait+0x7f/0xf0 [45770.713527] [] ? add_wait_queue+0x3c/0x50 [45770.719874] [] schedule_hrtimeout_range+0x13/0x20 [45770.726981] [] poll_schedule_timeout+0x55/0xc0 [45770.733799] [] do_sys_poll+0x48d/0x590 [45770.739847] [] ? unix_stream_recvmsg+0x54/0x70 [45770.746672] [] ? unix_ioctl+0x70/0x70 [45770.752659] [] ? sock_recvmsg+0xc5/0x100 [45770.758902] [] ? __enqueue_entity+0x78/0x80 [45770.765436] [] ? __pollwait+0xf0/0xf0 [45770.771375] [] ? __pollwait+0xf0/0xf0 [45770.777320] [] ? __pollwait+0xf0/0xf0 [45770.783279] [] ? try_to_wake_up+0x192/0x390 [45770.789810] [] ? default_wake_function+0x12/0x20 [45770.796825] [] ? pollwake+0x73/0x90 [45770.802590] [] ? wake_up_state+0x20/0x20 [45770.808832] [] ? __wake_up_common+0x82/0x120 [45770.815460] [] ? eventfd_ctx_read+0x67/0x1e0 [45770.822116] [] ? system_call_after_swapgs+0xa2/0x13a [45770.829525] [] SyS_poll+0x74/0x110 [45770.835188] [] ? system_call_after_swapgs+0x96/0x13a [45770.842606] [] ? system_call_after_swapgs+0xa2/0x13a [45770.850013] [] system_call_fastpath+0x25/0x2a [45770.856732] [] ? system_call_after_swapgs+0xa2/0x13a [45770.864136] irqbalance S ffff99ec6b433180 0 1101 1 0x00000080 [45770.872089] Call Trace: [45770.874836] [] schedule+0x29/0x70 [45770.880408] [] do_nanosleep+0x70/0x100 [45770.886445] [] hrtimer_nanosleep+0xbb/0x180 [45770.892954] [] ? hrtimer_get_res+0x50/0x50 [45770.899392] [] ? do_nanosleep+0x5b/0x100 [45770.905652] [] SyS_nanosleep+0x96/0xb0 [45770.911714] [] system_call_fastpath+0x25/0x2a [45770.918436] [] ? system_call_after_swapgs+0xa2/0x13a [45770.925846] sssd S ffff99ec69418000 0 1104 1 0x00000080 [45770.933798] Call Trace: [45770.936545] [] schedule+0x29/0x70 [45770.942113] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45770.949908] [] ? hrtimer_get_res+0x50/0x50 [45770.956348] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45770.964345] [] schedule_hrtimeout_range+0x13/0x20 [45770.971470] [] poll_schedule_timeout+0x55/0xc0 [45770.978296] [] do_sys_poll+0x48d/0x590 [45770.984363] [] ? sock_recvmsg+0xc5/0x100 [45770.990610] [] ? __pollwait+0xf0/0xf0 [45770.996546] [] ? __pollwait+0xf0/0xf0 [45771.002499] [] ? __pollwait+0xf0/0xf0 [45771.008450] [] ? __pollwait+0xf0/0xf0 [45771.014395] [] ? __pollwait+0xf0/0xf0 [45771.020351] [] ? __pollwait+0xf0/0xf0 [45771.026296] [] ? __pollwait+0xf0/0xf0 [45771.032254] [] ? __pollwait+0xf0/0xf0 [45771.038210] [] ? __pollwait+0xf0/0xf0 [45771.044168] [] SyS_poll+0x74/0x110 [45771.049825] [] system_call_fastpath+0x25/0x2a [45771.056546] [] ? system_call_after_swapgs+0xa2/0x13a [45771.063965] mcelog S ffff99ec6b5a1660 0 1141 1 0x00000080 [45771.071902] Call Trace: [45771.074671] [] schedule+0x29/0x70 [45771.080227] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45771.088126] [] ? add_wait_queue+0x3c/0x50 [45771.094483] [] ? add_wait_queue+0x3c/0x50 [45771.100831] [] schedule_hrtimeout_range+0x13/0x20 [45771.107948] [] poll_schedule_timeout+0x55/0xc0 [45771.114765] [] do_sys_poll+0x48d/0x590 [45771.120823] [] ? shmem_add_to_page_cache.isra.26+0xd0/0x120 [45771.128912] [] ? shmem_getpage_gfp+0x71f/0x860 [45771.135756] [] ? __radix_tree_lookup+0x84/0xf0 [45771.142581] [] ? __pollwait+0xf0/0xf0 [45771.148531] [] ? __pollwait+0xf0/0xf0 [45771.154507] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45771.161923] [] ? __do_fault.isra.61+0x8a/0x100 [45771.168771] [] ? unlock_page+0x2b/0x30 [45771.174823] [] ? do_read_fault.isra.63+0x139/0x1b0 [45771.182037] [] ? handle_mm_fault+0xa20/0xfb0 [45771.188689] [] ? recalc_sigpending+0x1b/0x70 [45771.195324] [] ? __set_task_blocked+0x41/0xa0 [45771.202053] [] SyS_ppoll+0xee/0x1f0 [45771.207816] [] ? system_call_after_swapgs+0x96/0x13a [45771.215224] [] ? system_call_after_swapgs+0xa2/0x13a [45771.222638] [] ? system_call_after_swapgs+0x96/0x13a [45771.230062] [] ? system_call_after_swapgs+0xa2/0x13a [45771.237470] [] system_call_fastpath+0x25/0x2a [45771.244200] [] ? system_call_after_swapgs+0xa2/0x13a [45771.251607] gssproxy S ffff99ec6ac23180 0 1175 1 0x00000080 [45771.259538] Call Trace: [45771.262293] [] schedule+0x29/0x70 [45771.267851] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45771.275666] [] ? hrtimer_get_res+0x50/0x50 [45771.282112] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45771.290102] [] schedule_hrtimeout_range+0x13/0x20 [45771.297226] [] ep_poll+0x23e/0x360 [45771.302880] [] ? wake_up_state+0x20/0x20 [45771.309123] [] SyS_epoll_wait+0xed/0x120 [45771.315361] [] ? system_call_after_swapgs+0x96/0x13a [45771.322765] [] system_call_fastpath+0x25/0x2a [45771.329501] [] ? system_call_after_swapgs+0xa2/0x13a [45771.336913] gssproxy S ffff99ec6ba39660 0 1187 1 0x00000080 [45771.344854] Call Trace: [45771.347605] [] schedule+0x29/0x70 [45771.353179] [] futex_wait_queue_me+0xc6/0x130 [45771.359913] [] futex_wait+0x17b/0x280 [45771.365867] [] ? __do_fault.isra.61+0x8a/0x100 [45771.372695] [] ? unlock_page+0x2b/0x30 [45771.378775] [] ? do_read_fault.isra.63+0x139/0x1b0 [45771.385979] [] do_futex+0x106/0x4d0 [45771.391743] [] SyS_futex+0x80/0x190 [45771.397511] [] ? system_call_after_swapgs+0x96/0x13a [45771.404943] [] ? system_call_after_swapgs+0xa2/0x13a [45771.412358] [] ? system_call_after_swapgs+0x96/0x13a [45771.419766] [] ? system_call_after_swapgs+0xa2/0x13a [45771.427174] [] ? system_call_after_swapgs+0x96/0x13a [45771.434591] [] ? system_call_after_swapgs+0xa2/0x13a [45771.441992] [] system_call_fastpath+0x25/0x2a [45771.448722] [] ? system_call_after_swapgs+0xa2/0x13a [45771.456143] gssproxy S ffff99ec6ba3b760 0 1188 1 0x00000080 [45771.464085] Call Trace: [45771.466842] [] schedule+0x29/0x70 [45771.472421] [] futex_wait_queue_me+0xc6/0x130 [45771.479155] [] futex_wait+0x17b/0x280 [45771.485116] [] ? dequeue_entity+0x11c/0x5c0 [45771.491651] [] ? put_prev_entity+0x31/0x400 [45771.498199] [] ? pick_next_task_fair+0x1c6/0x870 [45771.505217] [] do_futex+0x106/0x4d0 [45771.510969] [] ? __free_pages+0x1d/0x30 [45771.517115] [] SyS_futex+0x80/0x190 [45771.522861] [] ? system_call_after_swapgs+0x96/0x13a [45771.530267] [] ? system_call_after_swapgs+0xa2/0x13a [45771.537706] [] ? system_call_after_swapgs+0x96/0x13a [45771.545106] [] ? system_call_after_swapgs+0xa2/0x13a [45771.552514] [] ? system_call_after_swapgs+0x96/0x13a [45771.559915] [] ? system_call_after_swapgs+0xa2/0x13a [45771.567322] [] system_call_fastpath+0x25/0x2a [45771.574055] [] ? system_call_after_swapgs+0xa2/0x13a [45771.581451] gssproxy S ffff99ec6ba385e0 0 1189 1 0x00000080 [45771.589397] Call Trace: [45771.592150] [] schedule+0x29/0x70 [45771.597714] [] futex_wait_queue_me+0xc6/0x130 [45771.604450] [] futex_wait+0x17b/0x280 [45771.610416] [] ? dequeue_entity+0x11c/0x5c0 [45771.616949] [] ? put_prev_entity+0x31/0x400 [45771.623483] [] ? pick_next_task_fair+0x1c6/0x870 [45771.630502] [] do_futex+0x106/0x4d0 [45771.636246] [] ? schedule+0x29/0x70 [45771.642003] [] ? do_exit+0x693/0xa30 [45771.647854] [] SyS_futex+0x80/0x190 [45771.653619] [] ? system_call_after_swapgs+0x96/0x13a [45771.661025] [] ? system_call_after_swapgs+0xa2/0x13a [45771.668433] [] ? system_call_after_swapgs+0x96/0x13a [45771.675832] [] ? system_call_after_swapgs+0xa2/0x13a [45771.683239] [] ? system_call_after_swapgs+0x96/0x13a [45771.690640] [] ? system_call_after_swapgs+0xa2/0x13a [45771.698039] [] system_call_fastpath+0x25/0x2a [45771.704771] [] ? system_call_after_swapgs+0xa2/0x13a [45771.712182] gssproxy S ffff99ec6ba3c7e0 0 1190 1 0x00000080 [45771.720138] Call Trace: [45771.722884] [] schedule+0x29/0x70 [45771.728455] [] futex_wait_queue_me+0xc6/0x130 [45771.735182] [] futex_wait+0x17b/0x280 [45771.741126] [] ? dequeue_entity+0x11c/0x5c0 [45771.747660] [] ? put_prev_entity+0x31/0x400 [45771.754192] [] ? pick_next_task_fair+0x1c6/0x870 [45771.761196] [] do_futex+0x106/0x4d0 [45771.766955] [] ? schedule+0x29/0x70 [45771.772714] [] ? do_exit+0x693/0xa30 [45771.778576] [] SyS_futex+0x80/0x190 [45771.784335] [] ? system_call_after_swapgs+0x96/0x13a [45771.791754] [] ? system_call_after_swapgs+0xa2/0x13a [45771.799152] [] ? system_call_after_swapgs+0x96/0x13a [45771.806558] [] ? system_call_after_swapgs+0xa2/0x13a [45771.813956] [] ? system_call_after_swapgs+0x96/0x13a [45771.821365] [] ? system_call_after_swapgs+0xa2/0x13a [45771.828789] [] system_call_fastpath+0x25/0x2a [45771.835512] [] ? system_call_after_swapgs+0xa2/0x13a [45771.842918] gssproxy S ffff99ec6ba3e300 0 1191 1 0x00000080 [45771.850841] Call Trace: [45771.853602] [] schedule+0x29/0x70 [45771.859169] [] futex_wait_queue_me+0xc6/0x130 [45771.865903] [] futex_wait+0x17b/0x280 [45771.871855] [] ? __sb_end_write+0x31/0x70 [45771.878197] [] ? pipe_write+0x48f/0x540 [45771.884351] [] do_futex+0x106/0x4d0 [45771.890111] [] SyS_futex+0x80/0x190 [45771.895870] [] ? system_call_after_swapgs+0x96/0x13a [45771.903277] [] ? system_call_after_swapgs+0xa2/0x13a [45771.910704] [] ? system_call_after_swapgs+0x96/0x13a [45771.918111] [] ? system_call_after_swapgs+0xa2/0x13a [45771.925509] [] ? system_call_after_swapgs+0x96/0x13a [45771.932915] [] ? system_call_after_swapgs+0xa2/0x13a [45771.940321] [] system_call_fastpath+0x25/0x2a [45771.947055] [] ? system_call_after_swapgs+0xa2/0x13a [45771.954463] rpc.gssd S ffff99ec6a534200 0 1209 1 0x00000080 [45771.962433] Call Trace: [45771.965196] [] schedule+0x29/0x70 [45771.970776] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45771.978695] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [45771.986283] [] schedule_hrtimeout_range+0x13/0x20 [45771.993409] [] ep_poll+0x23e/0x360 [45771.999079] [] ? abort_exclusive_wait+0xa0/0xa0 [45772.006002] [] ? wake_up_state+0x20/0x20 [45772.012247] [] SyS_epoll_wait+0xed/0x120 [45772.018501] [] ? system_call_after_swapgs+0x96/0x13a [45772.025909] [] system_call_fastpath+0x25/0x2a [45772.032639] [] ? system_call_after_swapgs+0xa2/0x13a [45772.040047] sssd_be S ffff99e86e71c200 0 1257 1104 0x00000080 [45772.047977] Call Trace: [45772.050721] [] schedule+0x29/0x70 [45772.056280] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45772.064081] [] ? hrtimer_get_res+0x50/0x50 [45772.070520] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45772.078508] [] schedule_hrtimeout_range+0x13/0x20 [45772.085616] [] ep_poll+0x23e/0x360 [45772.091264] [] ? recalc_sigpending+0x1b/0x70 [45772.097884] [] ? wake_up_state+0x20/0x20 [45772.104133] [] SyS_epoll_wait+0xed/0x120 [45772.110377] [] ? system_call_after_swapgs+0x96/0x13a [45772.117804] [] system_call_fastpath+0x25/0x2a [45772.124539] [] ? system_call_after_swapgs+0xa2/0x13a [45772.131935] polkitd S ffff99e86e71e300 0 1291 1 0x00000080 [45772.139873] Call Trace: [45772.142619] [] schedule+0x29/0x70 [45772.148183] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45772.156062] [] ? add_wait_queue+0x3c/0x50 [45772.162406] [] schedule_hrtimeout_range+0x13/0x20 [45772.169532] [] poll_schedule_timeout+0x55/0xc0 [45772.176344] [] do_sys_poll+0x48d/0x590 [45772.182408] [] ? dequeue_entity+0x11c/0x5c0 [45772.188950] [] ? put_prev_entity+0x31/0x400 [45772.195484] [] ? __switch_to+0x151/0x580 [45772.201736] [] ? __enqueue_entity+0x78/0x80 [45772.208270] [] ? __pollwait+0xf0/0xf0 [45772.214230] [] ? __pollwait+0xf0/0xf0 [45772.220184] [] ? ttwu_do_activate+0x6f/0x80 [45772.226735] [] ? try_to_wake_up+0x192/0x390 [45772.233263] [] ? inotify_free_event+0xe/0x10 [45772.239896] [] ? kfree+0x106/0x140 [45772.245573] [] ? list_del+0xd/0x30 [45772.251222] [] ? remove_wait_queue+0x31/0x40 [45772.257852] [] ? eventfd_ctx_read+0x67/0x1e0 [45772.264485] [] ? system_call_after_swapgs+0xa2/0x13a [45772.271890] [] SyS_poll+0x74/0x110 [45772.277560] [] ? system_call_after_swapgs+0x96/0x13a [45772.284966] [] ? system_call_after_swapgs+0xa2/0x13a [45772.292376] [] system_call_fastpath+0x25/0x2a [45772.299097] [] ? system_call_after_swapgs+0xa2/0x13a [45772.306488] gmain S ffff99e6b19e05e0 0 1314 1 0x00000080 [45772.314456] Call Trace: [45772.317217] [] schedule+0x29/0x70 [45772.322796] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45772.330689] [] ? add_wait_queue+0x3c/0x50 [45772.337036] [] schedule_hrtimeout_range+0x13/0x20 [45772.344151] [] poll_schedule_timeout+0x55/0xc0 [45772.350974] [] do_sys_poll+0x48d/0x590 [45772.357033] [] ? __radix_tree_lookup+0x84/0xf0 [45772.363853] [] ? radix_tree_lookup_slot+0x22/0x50 [45772.370966] [] ? __find_get_page+0x1e/0xa0 [45772.377426] [] ? __pollwait+0xf0/0xf0 [45772.383385] [] ? __pollwait+0xf0/0xf0 [45772.389340] [] ? unlock_page+0x2b/0x30 [45772.395400] [] ? do_read_fault.isra.63+0x139/0x1b0 [45772.402616] [] ? handle_mm_fault+0xa20/0xfb0 [45772.409255] [] ? eventfd_ctx_read+0x67/0x1e0 [45772.415891] [] ? system_call_after_swapgs+0xa2/0x13a [45772.423298] [] SyS_poll+0x74/0x110 [45772.428962] [] ? system_call_after_swapgs+0x96/0x13a [45772.436370] [] ? system_call_after_swapgs+0xa2/0x13a [45772.443781] [] system_call_fastpath+0x25/0x2a [45772.450511] [] ? system_call_after_swapgs+0xa2/0x13a [45772.457914] gdbus S ffff99e6b19e1080 0 1315 1 0x00000080 [45772.465855] Call Trace: [45772.468613] [] schedule+0x29/0x70 [45772.474191] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45772.482086] [] ? add_wait_queue+0x3c/0x50 [45772.488434] [] ? add_wait_queue+0x3c/0x50 [45772.494789] [] schedule_hrtimeout_range+0x13/0x20 [45772.501912] [] poll_schedule_timeout+0x55/0xc0 [45772.508757] [] do_sys_poll+0x48d/0x590 [45772.514816] [] ? sock_sendmsg+0xb6/0xf0 [45772.520962] [] ? sock_recvmsg+0xc5/0x100 [45772.527206] [] ? __enqueue_entity+0x78/0x80 [45772.533751] [] ? __pollwait+0xf0/0xf0 [45772.539698] [] ? __pollwait+0xf0/0xf0 [45772.545674] [] ? ___sys_sendmsg+0x3e9/0x400 [45772.552208] [] ? default_wake_function+0x12/0x20 [45772.559234] [] ? pollwake+0x73/0x90 [45772.564986] [] ? wake_up_state+0x20/0x20 [45772.571214] [] ? __wake_up_common+0x82/0x120 [45772.577841] [] ? eventfd_ctx_read+0x67/0x1e0 [45772.584470] [] ? system_call_after_swapgs+0xa2/0x13a [45772.591879] [] SyS_poll+0x74/0x110 [45772.597540] [] ? system_call_after_swapgs+0x96/0x13a [45772.604948] [] ? system_call_after_swapgs+0xa2/0x13a [45772.612357] [] system_call_fastpath+0x25/0x2a [45772.619086] [] ? system_call_after_swapgs+0xa2/0x13a [45772.626507] polkitd S ffff99e6b19e26e0 0 1316 1 0x00000080 [45772.634485] Call Trace: [45772.637258] [] schedule+0x29/0x70 [45772.642826] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45772.650731] [] ? add_wait_queue+0x3c/0x50 [45772.657064] [] schedule_hrtimeout_range+0x13/0x20 [45772.664179] [] poll_schedule_timeout+0x55/0xc0 [45772.670997] [] do_sys_poll+0x48d/0x590 [45772.677063] [] ? __alloc_pages_nodemask+0x184/0xbe0 [45772.684374] [] ? __pollwait+0xf0/0xf0 [45772.690342] [] ? sched_clock+0x9/0x10 [45772.696288] [] ? sched_clock_cpu+0x85/0xc0 [45772.702746] [] ? try_to_wake_up+0x192/0x390 [45772.709272] [] ? wake_up_q+0x5b/0x80 [45772.715130] [] ? futex_wake+0x90/0x180 [45772.721167] [] ? eventfd_ctx_read+0x67/0x1e0 [45772.727822] [] ? system_call_after_swapgs+0xa2/0x13a [45772.735228] [] SyS_poll+0x74/0x110 [45772.740883] [] ? system_call_after_swapgs+0x96/0x13a [45772.748291] [] ? system_call_after_swapgs+0xa2/0x13a [45772.755700] [] system_call_fastpath+0x25/0x2a [45772.762430] [] ? system_call_after_swapgs+0xa2/0x13a [45772.769826] JS GC Helper S ffff99e6b19e47e0 0 1317 1 0x00000080 [45772.777775] Call Trace: [45772.780547] [] schedule+0x29/0x70 [45772.786099] [] futex_wait_queue_me+0xc6/0x130 [45772.792827] [] futex_wait+0x17b/0x280 [45772.798804] [] ? unlock_page+0x2b/0x30 [45772.804847] [] ? do_read_fault.isra.63+0x139/0x1b0 [45772.812052] [] do_futex+0x106/0x4d0 [45772.817820] [] SyS_futex+0x80/0x190 [45772.823619] [] ? system_call_after_swapgs+0x96/0x13a [45772.831014] [] ? system_call_after_swapgs+0xa2/0x13a [45772.838428] [] ? system_call_after_swapgs+0x96/0x13a [45772.845830] [] ? system_call_after_swapgs+0xa2/0x13a [45772.853246] [] ? system_call_after_swapgs+0x96/0x13a [45772.860645] [] ? system_call_after_swapgs+0xa2/0x13a [45772.868039] [] system_call_fastpath+0x25/0x2a [45772.874767] [] ? system_call_after_swapgs+0xa2/0x13a [45772.882180] JS Sour~ Thread S ffff99e6b19e5860 0 1318 1 0x00000080 [45772.890143] Call Trace: [45772.892887] [] schedule+0x29/0x70 [45772.898454] [] futex_wait_queue_me+0xc6/0x130 [45772.905181] [] futex_wait+0x17b/0x280 [45772.911137] [] ? sched_clock_cpu+0x85/0xc0 [45772.917577] [] ? try_to_wake_up+0x192/0x390 [45772.924104] [] do_futex+0x106/0x4d0 [45772.929864] [] ? cap_task_prctl+0x288/0x390 [45772.936386] [] ? yama_task_prctl+0x24/0x110 [45772.942912] [] ? cap_task_prctl+0x288/0x390 [45772.949454] [] ? strlcpy+0x42/0x60 [45772.955106] [] SyS_futex+0x80/0x190 [45772.960867] [] ? system_call_after_swapgs+0x96/0x13a [45772.968275] [] ? system_call_after_swapgs+0xa2/0x13a [45772.975676] [] ? system_call_after_swapgs+0x96/0x13a [45772.983084] [] ? system_call_after_swapgs+0xa2/0x13a [45772.990490] [] ? system_call_after_swapgs+0x96/0x13a [45772.997890] [] ? system_call_after_swapgs+0xa2/0x13a [45773.005298] [] system_call_fastpath+0x25/0x2a [45773.012034] [] ? system_call_after_swapgs+0xa2/0x13a [45773.019447] polkitd S ffff99ec651185e0 0 1319 1 0x00000080 [45773.027410] Call Trace: [45773.030170] [] schedule+0x29/0x70 [45773.035751] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45773.043635] [] ? add_wait_queue+0x3c/0x50 [45773.049978] [] schedule_hrtimeout_range+0x13/0x20 [45773.057087] [] poll_schedule_timeout+0x55/0xc0 [45773.063913] [] do_sys_poll+0x48d/0x590 [45773.069981] [] ? get_futex_key+0x1c8/0x2c0 [45773.076432] [] ? __pollwait+0xf0/0xf0 [45773.082384] [] ? futex_wait+0x162/0x280 [45773.088544] [] ? sched_clock_cpu+0x85/0xc0 [45773.094975] [] ? try_to_wake_up+0x192/0x390 [45773.101502] [] ? get_futex_key+0x1c8/0x2c0 [45773.107932] [] ? futex_wake+0x90/0x180 [45773.113974] [] ? do_futex+0x12a/0x4d0 [45773.119934] [] ? system_call_after_swapgs+0xa2/0x13a [45773.127347] [] SyS_poll+0x74/0x110 [45773.133009] [] ? system_call_after_swapgs+0x96/0x13a [45773.140417] [] ? system_call_after_swapgs+0xa2/0x13a [45773.147832] [] system_call_fastpath+0x25/0x2a [45773.154569] [] ? system_call_after_swapgs+0xa2/0x13a [45773.161980] dhclient S ffff99e840e78000 0 1299 1100 0x00000080 [45773.169931] Call Trace: [45773.172684] [] schedule+0x29/0x70 [45773.178242] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45773.186028] [] ? hrtimer_get_res+0x50/0x50 [45773.192472] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45773.200453] [] schedule_hrtimeout_range+0x13/0x20 [45773.207567] [] poll_schedule_timeout+0x55/0xc0 [45773.214405] [] do_select+0x6d1/0x7c0 [45773.220254] [] ? __pollwait+0xf0/0xf0 [45773.226198] [] ? __pollwait+0xf0/0xf0 [45773.232146] [] ? memcpy_toiovec+0x5c/0xb0 [45773.238491] [] ? skb_release_data+0x142/0x150 [45773.245225] [] ? kfree_skbmem+0x37/0x90 [45773.251363] [] ? consume_skb+0x34/0x90 [45773.257411] [] ? skb_consume_udp+0x23/0x80 [45773.263848] [] ? udp_recvmsg+0x2a4/0x480 [45773.270093] [] ? inet_recvmsg+0x80/0xb0 [45773.276241] [] ? sock_recvmsg+0xc5/0x100 [45773.282486] [] core_sys_select+0x1ff/0x340 [45773.288925] [] ? move_addr_to_user+0xa1/0xe0 [45773.295563] [] ? SYSC_recvfrom+0x10f/0x160 [45773.301996] [] ? ktime_get_ts64+0x52/0xf0 [45773.308329] [] SyS_select+0xca/0x120 [45773.314185] [] system_call_fastpath+0x25/0x2a [45773.320901] [] ? system_call_after_swapgs+0xa2/0x13a [45773.328306] sssd_nss S ffff99e840e79080 0 1302 1104 0x00000080 [45773.336256] Call Trace: [45773.338994] [] schedule+0x29/0x70 [45773.344565] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45773.352358] [] ? hrtimer_get_res+0x50/0x50 [45773.358796] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45773.366787] [] schedule_hrtimeout_range+0x13/0x20 [45773.373897] [] ep_poll+0x23e/0x360 [45773.379553] [] ? recalc_sigpending+0x1b/0x70 [45773.386183] [] ? wake_up_state+0x20/0x20 [45773.392425] [] SyS_epoll_wait+0xed/0x120 [45773.398684] [] ? system_call_after_swapgs+0x96/0x13a [45773.406078] [] system_call_fastpath+0x25/0x2a [45773.412806] [] ? system_call_after_swapgs+0xa2/0x13a [45773.420221] sssd_pam S ffff99e840e7a100 0 1303 1104 0x00000080 [45773.428145] Call Trace: [45773.430892] [] schedule+0x29/0x70 [45773.436450] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45773.444244] [] ? hrtimer_get_res+0x50/0x50 [45773.450681] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45773.458676] [] schedule_hrtimeout_range+0x13/0x20 [45773.465786] [] ep_poll+0x23e/0x360 [45773.471448] [] ? recalc_sigpending+0x1b/0x70 [45773.478071] [] ? wake_up_state+0x20/0x20 [45773.484321] [] SyS_epoll_wait+0xed/0x120 [45773.490564] [] ? system_call_after_swapgs+0x96/0x13a [45773.497970] [] system_call_fastpath+0x25/0x2a [45773.504698] [] ? system_call_after_swapgs+0xa2/0x13a [45773.512103] sssd_ssh S ffff99e840e7b180 0 1304 1104 0x00000080 [45773.520038] Call Trace: [45773.522796] [] schedule+0x29/0x70 [45773.528368] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45773.536150] [] ? hrtimer_get_res+0x50/0x50 [45773.542596] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45773.550586] [] schedule_hrtimeout_range+0x13/0x20 [45773.557703] [] ep_poll+0x23e/0x360 [45773.563357] [] ? recalc_sigpending+0x1b/0x70 [45773.569989] [] ? wake_up_state+0x20/0x20 [45773.576227] [] SyS_epoll_wait+0xed/0x120 [45773.582472] [] ? system_call_after_swapgs+0x96/0x13a [45773.589886] [] system_call_fastpath+0x25/0x2a [45773.596622] [] ? system_call_after_swapgs+0xa2/0x13a [45773.604028] sssd_pac S ffff99e840e7c200 0 1305 1104 0x00000080 [45773.611986] Call Trace: [45773.614735] [] schedule+0x29/0x70 [45773.620301] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45773.628088] [] ? hrtimer_get_res+0x50/0x50 [45773.634524] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45773.642513] [] schedule_hrtimeout_range+0x13/0x20 [45773.649644] [] ep_poll+0x23e/0x360 [45773.655314] [] ? recalc_sigpending+0x1b/0x70 [45773.661953] [] ? wake_up_state+0x20/0x20 [45773.668200] [] SyS_epoll_wait+0xed/0x120 [45773.674445] [] ? system_call_after_swapgs+0x96/0x13a [45773.681863] [] system_call_fastpath+0x25/0x2a [45773.688594] [] ? system_call_after_swapgs+0xa2/0x13a [45773.695986] mlx4_en S ffff99e6b19e3760 0 1307 2 0x00000080 [45773.703941] Call Trace: [45773.706710] [] schedule+0x29/0x70 [45773.712278] [] rescuer_thread+0x31e/0x3c0 [45773.718626] [] ? __schedule+0x320/0x680 [45773.724794] [] ? process_one_work+0x440/0x440 [45773.731521] [] kthread+0xd1/0xe0 [45773.736989] [] ? insert_kthread_work+0x40/0x40 [45773.743809] [] ret_from_fork_nospec_begin+0x21/0x21 [45773.751097] [] ? insert_kthread_work+0x40/0x40 [45773.757922] systemd-logind S ffff99e840e7e300 0 1312 1 0x00000080 [45773.765850] Call Trace: [45773.768602] [] schedule+0x29/0x70 [45773.774167] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45773.782060] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [45773.789670] [] schedule_hrtimeout_range+0x13/0x20 [45773.796806] [] ep_poll+0x23e/0x360 [45773.802468] [] ? SyS_epoll_ctl+0x24d/0xc80 [45773.808905] [] ? wake_up_state+0x20/0x20 [45773.815150] [] SyS_epoll_wait+0xed/0x120 [45773.821397] [] ? system_call_after_swapgs+0x96/0x13a [45773.828842] [] system_call_fastpath+0x25/0x2a [45773.835572] [] ? system_call_after_swapgs+0xa2/0x13a [45773.842991] mlx5_ib_sigerr_ S ffff99ec6511a100 0 1363 2 0x00000080 [45773.850935] Call Trace: [45773.853701] [] schedule+0x29/0x70 [45773.859269] [] rescuer_thread+0x31e/0x3c0 [45773.865611] [] ? __schedule+0x320/0x680 [45773.871772] [] ? process_one_work+0x440/0x440 [45773.878499] [] kthread+0xd1/0xe0 [45773.883969] [] ? insert_kthread_work+0x40/0x40 [45773.890802] [] ret_from_fork_nospec_begin+0x21/0x21 [45773.898119] [] ? insert_kthread_work+0x40/0x40 [45773.904943] ipoib_wq S ffff99ec6511b180 0 1391 2 0x00000080 [45773.912911] Call Trace: [45773.915670] [] schedule+0x29/0x70 [45773.921228] [] rescuer_thread+0x31e/0x3c0 [45773.927570] [] ? __schedule+0x320/0x680 [45773.933719] [] ? process_one_work+0x440/0x440 [45773.940449] [] kthread+0xd1/0xe0 [45773.945910] [] ? insert_kthread_work+0x40/0x40 [45773.952730] [] ret_from_fork_nospec_begin+0x21/0x21 [45773.960028] [] ? insert_kthread_work+0x40/0x40 [45773.966838] ipoib_wq S ffff99ec6511c200 0 1392 2 0x00000080 [45773.974756] Call Trace: [45773.977518] [] schedule+0x29/0x70 [45773.983091] [] rescuer_thread+0x31e/0x3c0 [45773.989437] [] ? __schedule+0x320/0x680 [45773.995583] [] ? process_one_work+0x440/0x440 [45774.002305] [] kthread+0xd1/0xe0 [45774.007792] [] ? insert_kthread_work+0x40/0x40 [45774.014602] [] ret_from_fork_nospec_begin+0x21/0x21 [45774.021899] [] ? insert_kthread_work+0x40/0x40 [45774.028708] rdma_cm S ffff99ec5d1aa100 0 1476 2 0x00000080 [45774.036650] Call Trace: [45774.039390] [] schedule+0x29/0x70 [45774.044941] [] rescuer_thread+0x31e/0x3c0 [45774.051267] [] ? __schedule+0x320/0x680 [45774.057403] [] ? process_one_work+0x440/0x440 [45774.064117] [] kthread+0xd1/0xe0 [45774.069594] [] ? insert_kthread_work+0x40/0x40 [45774.076421] [] ret_from_fork_nospec_begin+0x21/0x21 [45774.083723] [] ? insert_kthread_work+0x40/0x40 [45774.090563] kworker/30:2 S ffff99ec5d1a9080 0 1528 2 0x00000080 [45774.098533] Call Trace: [45774.101279] [] schedule+0x29/0x70 [45774.106832] [] worker_thread+0x1d9/0x3c0 [45774.113064] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45774.120361] [] kthread+0xd1/0xe0 [45774.125814] [] ? insert_kthread_work+0x40/0x40 [45774.132626] [] ret_from_fork_nospec_begin+0x21/0x21 [45774.139922] [] ? insert_kthread_work+0x40/0x40 [45774.146732] kworker/23:1H S ffff99e83c2c0000 0 1538 2 0x00000080 [45774.154656] Call Trace: [45774.157397] [] schedule+0x29/0x70 [45774.162948] [] worker_thread+0x1d9/0x3c0 [45774.169179] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45774.176475] [] kthread+0xd1/0xe0 [45774.181930] [] ? insert_kthread_work+0x40/0x40 [45774.188735] [] ret_from_fork_nospec_begin+0x21/0x21 [45774.196036] [] ? insert_kthread_work+0x40/0x40 [45774.202846] sshd S ffff99ec6bfba100 0 1801 1 0x00000080 [45774.210767] Call Trace: [45774.213507] [] schedule+0x29/0x70 [45774.219057] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45774.226935] [] ? add_wait_queue+0x3c/0x50 [45774.233263] [] ? __pollwait+0x7f/0xf0 [45774.239202] [] schedule_hrtimeout_range+0x13/0x20 [45774.246296] [] poll_schedule_timeout+0x55/0xc0 [45774.253109] [] do_select+0x6d1/0x7c0 [45774.258952] [] ? __pollwait+0xf0/0xf0 [45774.264892] [] ? __pollwait+0xf0/0xf0 [45774.270830] [] ? __pollwait+0xf0/0xf0 [45774.276770] [] ? __check_object_size+0x1ca/0x250 [45774.283778] [] ? memcg_check_events+0xbe/0x210 [45774.290608] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [45774.298187] [] ? cpumask_any_but+0x39/0x50 [45774.304634] [] ? flush_tlb_page+0x38/0xa0 [45774.310962] [] ? lru_cache_add+0xe/0x10 [45774.317097] [] ? page_add_new_anon_rmap+0xb8/0x170 [45774.324297] [] ? wp_page_copy.isra.73+0x3d5/0x5b0 [45774.331400] [] ? __check_object_size+0x1ca/0x250 [45774.338412] [] core_sys_select+0x1ff/0x340 [45774.344835] [] ? __d_free+0x35/0x40 [45774.350593] [] ? d_free+0x60/0x70 [45774.356145] [] ? __dentry_kill+0x160/0x1d0 [45774.362580] [] ? mntput+0x24/0x40 [45774.368136] [] ? __fput+0x195/0x230 [45774.373887] [] SyS_select+0xca/0x120 [45774.379730] [] ? system_call_after_swapgs+0x96/0x13a [45774.387115] [] ? system_call_after_swapgs+0xa2/0x13a [45774.394502] [] ? system_call_after_swapgs+0x96/0x13a [45774.401896] [] ? system_call_after_swapgs+0xa2/0x13a [45774.409292] [] system_call_fastpath+0x25/0x2a [45774.416006] [] ? system_call_after_swapgs+0xa2/0x13a [45774.423407] xinetd S ffff99ec69c29080 0 1806 1 0x00000080 [45774.431309] Call Trace: [45774.434049] [] schedule+0x29/0x70 [45774.439600] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45774.447499] [] ? try_to_wake_up+0x192/0x390 [45774.454033] [] ? add_wait_queue+0x3c/0x50 [45774.460374] [] schedule_hrtimeout_range+0x13/0x20 [45774.467483] [] poll_schedule_timeout+0x55/0xc0 [45774.474307] [] do_sys_poll+0x48d/0x590 [45774.480374] [] ? __wake_up_common_lock+0x91/0xc0 [45774.487398] [] ? task_rq_unlock+0x20/0x20 [45774.493745] [] ? __radix_tree_lookup+0x84/0xf0 [45774.500563] [] ? radix_tree_lookup_slot+0x22/0x50 [45774.507679] [] ? __find_get_page+0x1e/0xa0 [45774.514110] [] ? __pollwait+0xf0/0xf0 [45774.520083] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45774.527496] [] ? __do_fault.isra.61+0x8a/0x100 [45774.534323] [] ? unlock_page+0x2b/0x30 [45774.540380] [] ? do_read_fault.isra.63+0x139/0x1b0 [45774.547592] [] ? handle_mm_fault+0xa20/0xfb0 [45774.554238] [] ? __do_page_fault+0x238/0x500 [45774.560879] [] ? system_call_after_swapgs+0xa2/0x13a [45774.568286] [] SyS_poll+0x74/0x110 [45774.573949] [] ? system_call_after_swapgs+0x96/0x13a [45774.581357] [] ? system_call_after_swapgs+0xa2/0x13a [45774.588768] [] system_call_fastpath+0x25/0x2a [45774.595492] [] ? system_call_after_swapgs+0xa2/0x13a [45774.602907] munged S ffff99ec69699080 0 1808 1 0x00000080 [45774.610878] Call Trace: [45774.613658] [] schedule+0x29/0x70 [45774.619224] [] schedule_timeout+0x221/0x2d0 [45774.625762] [] ? prepare_to_wait_exclusive+0x56/0x90 [45774.633149] [] __skb_wait_for_more_packets+0x120/0x180 [45774.640752] [] ? datagram_poll+0x110/0x110 [45774.647176] [] __skb_recv_datagram+0x6f/0xd0 [45774.653824] [] skb_recv_datagram+0x43/0x60 [45774.660259] [] unix_accept+0x97/0x170 [45774.666198] [] SYSC_accept4+0x109/0x230 [45774.672353] [] ? __sb_end_write+0x31/0x70 [45774.678701] [] ? vfs_write+0x182/0x1f0 [45774.684758] [] ? system_call_after_swapgs+0xa2/0x13a [45774.692158] [] ? system_call_after_swapgs+0x96/0x13a [45774.699573] [] ? system_call_after_swapgs+0xa2/0x13a [45774.706973] [] ? system_call_after_swapgs+0x96/0x13a [45774.714381] [] ? system_call_after_swapgs+0xa2/0x13a [45774.721793] [] SyS_accept4+0xe/0x10 [45774.727568] [] SyS_accept+0x10/0x20 [45774.733326] [] system_call_fastpath+0x25/0x2a [45774.740046] [] ? system_call_after_swapgs+0xa2/0x13a [45774.747460] munged S ffff99e86bba3180 0 1829 1 0x00000080 [45774.755432] Call Trace: [45774.758177] [] schedule+0x29/0x70 [45774.763744] [] futex_wait_queue_me+0xc6/0x130 [45774.770465] [] futex_wait+0x17b/0x280 [45774.776416] [] ? hrtimer_get_res+0x50/0x50 [45774.782852] [] ? futex_wait_queue_me+0xa4/0x130 [45774.789768] [] do_futex+0x106/0x4d0 [45774.795519] [] ? mntput+0x24/0x40 [45774.801073] [] ? __fput+0x195/0x230 [45774.806864] [] SyS_futex+0x80/0x190 [45774.812615] [] system_call_fastpath+0x25/0x2a [45774.819346] [] ? system_call_after_swapgs+0xa2/0x13a [45774.826910] munged S ffff99e86bba5280 0 1830 1 0x00000080 [45774.834899] Call Trace: [45774.837646] [] schedule+0x29/0x70 [45774.843207] [] futex_wait_queue_me+0xc6/0x130 [45774.849936] [] futex_wait+0x17b/0x280 [45774.855903] [] do_futex+0x106/0x4d0 [45774.861663] [] ? recalc_sigpending+0x1b/0x70 [45774.868295] [] ? __set_task_blocked+0x41/0xa0 [45774.875030] [] SyS_futex+0x80/0x190 [45774.880783] [] ? system_call_after_swapgs+0x96/0x13a [45774.888192] [] ? system_call_after_swapgs+0xa2/0x13a [45774.895605] [] ? system_call_after_swapgs+0x96/0x13a [45774.903012] [] ? system_call_after_swapgs+0xa2/0x13a [45774.910419] [] ? system_call_after_swapgs+0x96/0x13a [45774.917836] [] ? system_call_after_swapgs+0xa2/0x13a [45774.925242] [] system_call_fastpath+0x25/0x2a [45774.931978] [] ? system_call_after_swapgs+0xa2/0x13a [45774.939380] munged S ffff99e86bba6300 0 1831 1 0x00000080 [45774.947339] Call Trace: [45774.950085] [] schedule+0x29/0x70 [45774.955652] [] futex_wait_queue_me+0xc6/0x130 [45774.962379] [] futex_wait+0x17b/0x280 [45774.968339] [] ? dequeue_entity+0x11c/0x5c0 [45774.974875] [] ? put_prev_entity+0x31/0x400 [45774.981422] [] ? pick_next_task_fair+0x1c6/0x870 [45774.988440] [] do_futex+0x106/0x4d0 [45774.994190] [] ? schedule+0x29/0x70 [45774.999955] [] ? recalc_sigpending+0x1b/0x70 [45775.006592] [] ? __set_task_blocked+0x41/0xa0 [45775.013327] [] SyS_futex+0x80/0x190 [45775.019078] [] ? system_call_after_swapgs+0x96/0x13a [45775.026477] [] ? system_call_after_swapgs+0xa2/0x13a [45775.033883] [] ? system_call_after_swapgs+0x96/0x13a [45775.041295] [] ? system_call_after_swapgs+0xa2/0x13a [45775.048703] [] ? system_call_after_swapgs+0x96/0x13a [45775.056097] [] ? system_call_after_swapgs+0xa2/0x13a [45775.063500] [] system_call_fastpath+0x25/0x2a [45775.070228] [] ? system_call_after_swapgs+0xa2/0x13a [45775.077633] rsyslogd S ffff99e86bba1080 0 1810 1 0x00000080 [45775.085573] Call Trace: [45775.088326] [] schedule+0x29/0x70 [45775.093903] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45775.101697] [] ? hrtimer_get_res+0x50/0x50 [45775.108131] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45775.116106] [] schedule_hrtimeout_range+0x13/0x20 [45775.123223] [] poll_schedule_timeout+0x55/0xc0 [45775.130055] [] do_select+0x6d1/0x7c0 [45775.135922] [] ? update_curr+0x14c/0x1e0 [45775.142166] [] ? enqueue_task_fair+0x208/0x6c0 [45775.148991] [] ? sched_clock_cpu+0x85/0xc0 [45775.155427] [] ? task_scan_min+0x3e/0x60 [45775.161662] [] ? task_scan_max+0x27/0x40 [45775.167917] [] ? task_numa_fault+0x908/0xc10 [45775.174557] [] ? do_numa_page+0x1be/0x250 [45775.180888] [] ? handle_mm_fault+0xadb/0xfb0 [45775.187519] [] ? should_numa_migrate_memory+0x5b/0x160 [45775.195121] [] ? mpol_misplaced+0x18c/0x1e0 [45775.201648] [] core_sys_select+0x1ff/0x340 [45775.208086] [] ? handle_mm_fault+0xadb/0xfb0 [45775.214710] [] ? ktime_get_ts64+0x52/0xf0 [45775.221036] [] SyS_select+0xca/0x120 [45775.226906] [] system_call_fastpath+0x25/0x2a [45775.233635] [] ? system_call_after_swapgs+0xa2/0x13a [45775.241034] in:imjournal S ffff99ec6acc3180 0 1839 1 0x00000080 [45775.248962] Call Trace: [45775.251707] [] schedule+0x29/0x70 [45775.257280] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45775.265081] [] ? hrtimer_get_res+0x50/0x50 [45775.271526] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45775.279516] [] schedule_hrtimeout_range+0x13/0x20 [45775.286631] [] poll_schedule_timeout+0x55/0xc0 [45775.293456] [] do_sys_poll+0x48d/0x590 [45775.299500] [] ? radix_tree_lookup_slot+0x22/0x50 [45775.306624] [] ? __find_get_page+0x1e/0xa0 [45775.313061] [] ? __enqueue_entity+0x78/0x80 [45775.319603] [] ? enqueue_entity+0x2ef/0xbe0 [45775.326146] [] ? link_path_walk+0x81/0x8b0 [45775.332582] [] ? __pollwait+0xf0/0xf0 [45775.338541] [] ? ttwu_do_wakeup+0xb5/0xe0 [45775.344881] [] ? ttwu_do_activate+0x6f/0x80 [45775.351407] [] ? try_to_wake_up+0x192/0x390 [45775.357943] [] ? free_pages_and_swap_cache+0xad/0xd0 [45775.365350] [] ? wake_up_q+0x5b/0x80 [45775.371203] [] ? futex_wake_op+0x3b5/0x640 [45775.377646] [] ? do_futex+0x17a/0x4d0 [45775.383598] [] ? ktime_get_ts64+0x52/0xf0 [45775.389951] [] SyS_ppoll+0x1d3/0x1f0 [45775.395810] [] ? system_call_after_swapgs+0x96/0x13a [45775.403205] [] ? system_call_after_swapgs+0xa2/0x13a [45775.410612] [] system_call_fastpath+0x25/0x2a [45775.417341] [] ? system_call_after_swapgs+0xa2/0x13a [45775.424737] rs:main Q:Reg S ffff99ec6accb180 0 1840 1 0x00000080 [45775.432701] Call Trace: [45775.435448] [] schedule+0x29/0x70 [45775.441007] [] futex_wait_queue_me+0xc6/0x130 [45775.447743] [] futex_wait+0x17b/0x280 [45775.453702] [] do_futex+0x106/0x4d0 [45775.459459] [] SyS_futex+0x80/0x190 [45775.465210] [] ? system_call_after_swapgs+0x96/0x13a [45775.472617] [] ? system_call_after_swapgs+0xa2/0x13a [45775.480021] [] ? system_call_after_swapgs+0x96/0x13a [45775.487434] [] ? system_call_after_swapgs+0xa2/0x13a [45775.494871] [] ? system_call_after_swapgs+0x96/0x13a [45775.502278] [] ? system_call_after_swapgs+0xa2/0x13a [45775.509692] [] system_call_fastpath+0x25/0x2a [45775.516414] [] ? system_call_after_swapgs+0xa2/0x13a [45775.523826] rpc.statd S ffff99ec6d825280 0 1815 1 0x00000080 [45775.531801] Call Trace: [45775.534552] [] schedule+0x29/0x70 [45775.540111] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45775.547997] [] ? add_wait_queue+0x3c/0x50 [45775.554338] [] ? __pollwait+0x7f/0xf0 [45775.560289] [] schedule_hrtimeout_range+0x13/0x20 [45775.567407] [] poll_schedule_timeout+0x55/0xc0 [45775.574225] [] do_select+0x6d1/0x7c0 [45775.580089] [] ? __slab_free+0x9d/0x290 [45775.586235] [] ? __pollwait+0xf0/0xf0 [45775.592187] [] ? __pollwait+0xf0/0xf0 [45775.598133] [] ? __pollwait+0xf0/0xf0 [45775.604078] [] ? __pollwait+0xf0/0xf0 [45775.610022] [] ? __pollwait+0xf0/0xf0 [45775.615972] [] ? radix_tree_lookup_slot+0x22/0x50 [45775.623096] [] ? __find_get_page+0x1e/0xa0 [45775.629537] [] ? filemap_fault+0x17d/0x420 [45775.635971] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45775.643380] [] ? __do_fault.isra.61+0x8a/0x100 [45775.650207] [] core_sys_select+0x1ff/0x340 [45775.656647] [] ? handle_mm_fault+0xa20/0xfb0 [45775.663278] [] ? __wake_up_common_lock+0x91/0xc0 [45775.670296] [] ? task_rq_unlock+0x20/0x20 [45775.676644] [] SyS_select+0xca/0x120 [45775.682502] [] ? system_call_after_swapgs+0x96/0x13a [45775.689910] [] ? system_call_after_swapgs+0xa2/0x13a [45775.697330] [] ? system_call_after_swapgs+0x96/0x13a [45775.704729] [] ? system_call_after_swapgs+0xa2/0x13a [45775.712137] [] system_call_fastpath+0x25/0x2a [45775.718875] [] ? system_call_after_swapgs+0xa2/0x13a [45775.726279] kworker/26:2 S ffff99ec6b975280 0 1822 2 0x00000080 [45775.734230] Call Trace: [45775.736984] [] schedule+0x29/0x70 [45775.742558] [] worker_thread+0x1d9/0x3c0 [45775.748802] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45775.756106] [] kthread+0xd1/0xe0 [45775.761587] [] ? insert_kthread_work+0x40/0x40 [45775.768423] [] ret_from_fork_nospec_begin+0x21/0x21 [45775.775721] [] ? insert_kthread_work+0x40/0x40 [45775.782554] nfsiod S ffff99ec6bfbd280 0 1849 2 0x00000080 [45775.790504] Call Trace: [45775.793257] [] schedule+0x29/0x70 [45775.798823] [] rescuer_thread+0x31e/0x3c0 [45775.805163] [] ? __schedule+0x320/0x680 [45775.811305] [] ? process_one_work+0x440/0x440 [45775.818033] [] kthread+0xd1/0xe0 [45775.823499] [] ? insert_kthread_work+0x40/0x40 [45775.830424] [] ret_from_fork_nospec_begin+0x21/0x21 [45775.837751] [] ? insert_kthread_work+0x40/0x40 [45775.844574] kworker/21:1H S ffff99e86b279080 0 1864 2 0x00000080 [45775.852524] Call Trace: [45775.855279] [] schedule+0x29/0x70 [45775.860834] [] worker_thread+0x1d9/0x3c0 [45775.867077] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45775.874386] [] kthread+0xd1/0xe0 [45775.879877] [] ? insert_kthread_work+0x40/0x40 [45775.886709] [] ret_from_fork_nospec_begin+0x21/0x21 [45775.894021] [] ? insert_kthread_work+0x40/0x40 [45775.900841] nfsv4.0-svc S ffff99e86b27b180 0 1874 2 0x00000080 [45775.908799] Call Trace: [45775.911568] [] schedule+0x29/0x70 [45775.917132] [] schedule_timeout+0x221/0x2d0 [45775.923672] [] ? kfree+0x106/0x140 [45775.929395] [] ? svc_xprt_free+0x77/0x90 [sunrpc] [45775.936543] [] svc_recv+0x980/0xa20 [sunrpc] [45775.943202] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [45775.951403] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [45775.959610] [] nfs4_callback_svc+0x33/0x60 [nfsv4] [45775.966833] [] kthread+0xd1/0xe0 [45775.972308] [] ? insert_kthread_work+0x40/0x40 [45775.979136] [] ret_from_fork_nospec_begin+0x21/0x21 [45775.986448] [] ? insert_kthread_work+0x40/0x40 [45775.993272] automount S ffff99e86bba0000 0 1924 1 0x00000080 [45776.001203] Call Trace: [45776.003960] [] schedule+0x29/0x70 [45776.009526] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45776.017426] [] ? sock_destroy_inode+0x2f/0x40 [45776.024155] [] ? recalc_sigpending+0x1b/0x70 [45776.030789] [] schedule_hrtimeout_range+0x13/0x20 [45776.037916] [] do_sigtimedwait+0x196/0x240 [45776.044356] [] SYSC_rt_sigtimedwait+0xb4/0x120 [45776.051174] [] ? mntput+0x24/0x40 [45776.056736] [] ? __fput+0x195/0x230 [45776.062495] [] ? system_call_after_swapgs+0xa2/0x13a [45776.069889] [] ? system_call_after_swapgs+0x96/0x13a [45776.077299] [] ? system_call_after_swapgs+0xa2/0x13a [45776.084713] [] ? system_call_after_swapgs+0x96/0x13a [45776.092112] [] ? system_call_after_swapgs+0xa2/0x13a [45776.099521] [] ? system_call_after_swapgs+0x96/0x13a [45776.106927] [] ? system_call_after_swapgs+0xa2/0x13a [45776.114351] [] ? system_call_after_swapgs+0x96/0x13a [45776.121762] [] ? system_call_after_swapgs+0xa2/0x13a [45776.129167] [] SyS_rt_sigtimedwait+0xe/0x10 [45776.135701] [] system_call_fastpath+0x25/0x2a [45776.142429] [] ? system_call_after_swapgs+0xa2/0x13a [45776.149843] automount S ffff99ec6d823180 0 2009 1 0x00000080 [45776.157821] Call Trace: [45776.160579] [] schedule+0x29/0x70 [45776.166144] [] futex_wait_queue_me+0xc6/0x130 [45776.172890] [] futex_wait+0x17b/0x280 [45776.178845] [] ? hrtimer_get_res+0x50/0x50 [45776.185283] [] ? futex_wait_queue_me+0xa4/0x130 [45776.192206] [] do_futex+0x106/0x4d0 [45776.197974] [] ? do_futex+0xdd/0x4d0 [45776.203823] [] SyS_futex+0x80/0x190 [45776.209595] [] system_call_fastpath+0x25/0x2a [45776.216317] [] ? system_call_after_swapgs+0xa2/0x13a [45776.223738] automount S ffff99ec6d821080 0 2010 1 0x00000080 [45776.231690] Call Trace: [45776.234449] [] schedule+0x29/0x70 [45776.240008] [] futex_wait_queue_me+0xc6/0x130 [45776.246735] [] futex_wait+0x17b/0x280 [45776.252693] [] ? sched_slice.isra.55+0x66/0xc0 [45776.259515] [] ? get_futex_key+0x1c8/0x2c0 [45776.265957] [] do_futex+0x106/0x4d0 [45776.271714] [] ? check_preempt_curr+0x80/0xa0 [45776.278441] [] ? wake_up_new_task+0x135/0x1a0 [45776.285162] [] SyS_futex+0x80/0x190 [45776.290915] [] ? system_call_after_swapgs+0x96/0x13a [45776.298324] [] ? system_call_after_swapgs+0xa2/0x13a [45776.305723] [] ? system_call_after_swapgs+0x96/0x13a [45776.313129] [] ? system_call_after_swapgs+0xa2/0x13a [45776.320527] [] ? system_call_after_swapgs+0x96/0x13a [45776.327933] [] ? system_call_after_swapgs+0xa2/0x13a [45776.335333] [] system_call_fastpath+0x25/0x2a [45776.342068] [] ? system_call_after_swapgs+0xa2/0x13a [45776.349478] automount S ffff99e83c3ab180 0 2165 1 0x00000080 [45776.357437] Call Trace: [45776.360182] [] schedule+0x29/0x70 [45776.365733] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45776.373625] [] ? add_wait_queue+0x3c/0x50 [45776.379972] [] schedule_hrtimeout_range+0x13/0x20 [45776.387090] [] poll_schedule_timeout+0x55/0xc0 [45776.393924] [] do_sys_poll+0x48d/0x590 [45776.399986] [] ? __d_lookup+0x7a/0x160 [45776.406034] [] ? mntput+0x24/0x40 [45776.411605] [] ? terminate_walk+0x49/0x50 [45776.417944] [] ? __radix_tree_lookup+0x84/0xf0 [45776.424784] [] ? radix_tree_lookup_slot+0x22/0x50 [45776.431919] [] ? __find_get_page+0x1e/0xa0 [45776.438355] [] ? __pollwait+0xf0/0xf0 [45776.444302] [] ? __pollwait+0xf0/0xf0 [45776.450247] [] ? __pollwait+0xf0/0xf0 [45776.456201] [] ? unlock_page+0x2b/0x30 [45776.462244] [] ? do_read_fault.isra.63+0x139/0x1b0 [45776.469462] [] ? handle_mm_fault+0xa20/0xfb0 [45776.476092] [] ? __do_page_fault+0x238/0x500 [45776.482725] [] ? system_call_after_swapgs+0xa2/0x13a [45776.490124] [] SyS_poll+0x74/0x110 [45776.495777] [] ? system_call_after_swapgs+0x96/0x13a [45776.503179] [] ? system_call_after_swapgs+0xa2/0x13a [45776.510571] [] system_call_fastpath+0x25/0x2a [45776.517299] [] ? system_call_after_swapgs+0xa2/0x13a [45776.524696] automount S ffff99e83c3aa100 0 2168 1 0x00000080 [45776.532646] Call Trace: [45776.535406] [] schedule+0x29/0x70 [45776.540978] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45776.548870] [] ? add_wait_queue+0x3c/0x50 [45776.555212] [] schedule_hrtimeout_range+0x13/0x20 [45776.562323] [] poll_schedule_timeout+0x55/0xc0 [45776.569150] [] do_sys_poll+0x48d/0x590 [45776.575200] [] ? __d_lookup+0x7a/0x160 [45776.581250] [] ? mntput+0x24/0x40 [45776.586821] [] ? terminate_walk+0x49/0x50 [45776.593161] [] ? __enqueue_entity+0x78/0x80 [45776.599703] [] ? enqueue_entity+0x2ef/0xbe0 [45776.606239] [] ? path_mountpoint+0x369/0x440 [45776.612870] [] ? __wake_up_common+0x82/0x120 [45776.619502] [] ? tracing_is_on+0x15/0x30 [45776.625731] [] ? tracing_record_cmdline+0x1d/0x120 [45776.632942] [] ? __pollwait+0xf0/0xf0 [45776.638908] [] ? __pollwait+0xf0/0xf0 [45776.644861] [] ? __pollwait+0xf0/0xf0 [45776.650816] [] ? futex_wake_op+0x3b5/0x640 [45776.657252] [] ? do_futex+0x17a/0x4d0 [45776.663212] [] ? autofs_dev_ioctl+0x13/0x20 [45776.669753] [] ? do_vfs_ioctl+0x3a0/0x5b0 [45776.676093] [] ? system_call_after_swapgs+0xa2/0x13a [45776.683510] [] SyS_poll+0x74/0x110 [45776.689172] [] ? system_call_after_swapgs+0x96/0x13a [45776.696579] [] ? system_call_after_swapgs+0xa2/0x13a [45776.703987] [] system_call_fastpath+0x25/0x2a [45776.710717] [] ? system_call_after_swapgs+0xa2/0x13a [45776.718128] automount S ffff99e83c3a9080 0 2172 1 0x00000080 [45776.726086] Call Trace: [45776.728832] [] schedule+0x29/0x70 [45776.734403] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45776.742289] [] ? add_wait_queue+0x3c/0x50 [45776.748636] [] schedule_hrtimeout_range+0x13/0x20 [45776.755739] [] poll_schedule_timeout+0x55/0xc0 [45776.762555] [] do_sys_poll+0x48d/0x590 [45776.768605] [] ? dequeue_entity+0x11c/0x5c0 [45776.775131] [] ? put_prev_entity+0x31/0x400 [45776.781671] [] ? __switch_to+0x151/0x580 [45776.787933] [] ? __schedule+0x320/0x680 [45776.794082] [] ? __pollwait+0xf0/0xf0 [45776.800040] [] ? __pollwait+0xf0/0xf0 [45776.806002] [] ? __pollwait+0xf0/0xf0 [45776.811951] [] ? hrtimer_get_res+0x50/0x50 [45776.818397] [] ? get_futex_key+0x1c8/0x2c0 [45776.824835] [] ? futex_wake+0x90/0x180 [45776.830875] [] ? do_futex+0x12a/0x4d0 [45776.836902] [] ? check_preempt_curr+0x80/0xa0 [45776.843624] [] ? wake_up_new_task+0x135/0x1a0 [45776.850360] [] ? system_call_after_swapgs+0xa2/0x13a [45776.857776] [] SyS_poll+0x74/0x110 [45776.863452] [] ? system_call_after_swapgs+0x96/0x13a [45776.870852] [] ? system_call_after_swapgs+0xa2/0x13a [45776.878258] [] system_call_fastpath+0x25/0x2a [45776.885002] [] ? system_call_after_swapgs+0xa2/0x13a [45776.892412] automount S ffff99e83c3ac200 0 2173 1 0x00000080 [45776.900343] Call Trace: [45776.903082] [] schedule+0x29/0x70 [45776.908648] [] schedule_hrtimeout_range_clock+0x12d/0x150 [45776.916541] [] ? handle_mm_fault+0xbfd/0xfb0 [45776.923179] [] ? __brelse+0x3d/0x50 [45776.928953] [] ? add_wait_queue+0x3c/0x50 [45776.935293] [] schedule_hrtimeout_range+0x13/0x20 [45776.942412] [] poll_schedule_timeout+0x55/0xc0 [45776.949236] [] do_sys_poll+0x48d/0x590 [45776.955280] [] ? __d_lookup+0x7a/0x160 [45776.961322] [] ? mntput+0x24/0x40 [45776.966881] [] ? terminate_walk+0x49/0x50 [45776.973221] [] ? __enqueue_entity+0x78/0x80 [45776.979756] [] ? enqueue_entity+0x2ef/0xbe0 [45776.986297] [] ? path_mountpoint+0x369/0x440 [45776.992951] [] ? __wake_up_common+0x82/0x120 [45776.999589] [] ? tracing_is_on+0x15/0x30 [45777.005819] [] ? tracing_record_cmdline+0x1d/0x120 [45777.013033] [] ? __pollwait+0xf0/0xf0 [45777.018999] [] ? __pollwait+0xf0/0xf0 [45777.024952] [] ? __pollwait+0xf0/0xf0 [45777.030932] [] ? futex_wake_op+0x3b5/0x640 [45777.037369] [] ? do_futex+0x17a/0x4d0 [45777.043320] [] ? autofs_dev_ioctl+0x13/0x20 [45777.049862] [] ? do_vfs_ioctl+0x3a0/0x5b0 [45777.056197] [] ? system_call_after_swapgs+0xa2/0x13a [45777.063604] [] SyS_poll+0x74/0x110 [45777.069265] [] ? system_call_after_swapgs+0x96/0x13a [45777.076675] [] ? system_call_after_swapgs+0xa2/0x13a [45777.084075] [] system_call_fastpath+0x25/0x2a [45777.090796] [] ? system_call_after_swapgs+0xa2/0x13a [45777.098202] master S ffff99ec6969e300 0 1925 1 0x00000080 [45777.106149] Call Trace: [45777.108929] [] schedule+0x29/0x70 [45777.114500] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45777.122295] [] ? hrtimer_get_res+0x50/0x50 [45777.128736] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45777.136724] [] schedule_hrtimeout_range+0x13/0x20 [45777.143836] [] ep_poll+0x23e/0x360 [45777.149505] [] ? wake_up_state+0x20/0x20 [45777.155735] [] SyS_epoll_wait+0xed/0x120 [45777.161991] [] ? system_call_after_swapgs+0x96/0x13a [45777.169405] [] system_call_fastpath+0x25/0x2a [45777.176142] [] ? system_call_after_swapgs+0xa2/0x13a [45777.183540] atd S ffff99e83c3a8000 0 1929 1 0x00000080 [45777.191485] Call Trace: [45777.194237] [] schedule+0x29/0x70 [45777.199801] [] do_nanosleep+0x70/0x100 [45777.205838] [] hrtimer_nanosleep+0xbb/0x180 [45777.212355] [] ? hrtimer_get_res+0x50/0x50 [45777.218793] [] ? do_nanosleep+0x5b/0x100 [45777.225028] [] SyS_nanosleep+0x96/0xb0 [45777.231066] [] system_call_fastpath+0x25/0x2a [45777.237820] [] ? system_call_after_swapgs+0xa2/0x13a [45777.245234] crond S ffff99e83c3ae300 0 1932 1 0x00000080 [45777.253194] Call Trace: [45777.255960] [] schedule+0x29/0x70 [45777.261524] [] do_nanosleep+0x70/0x100 [45777.267565] [] hrtimer_nanosleep+0xbb/0x180 [45777.274098] [] ? hrtimer_get_res+0x50/0x50 [45777.280524] [] ? do_nanosleep+0x5b/0x100 [45777.286773] [] SyS_nanosleep+0x96/0xb0 [45777.292817] [] system_call_fastpath+0x25/0x2a [45777.299544] [] ? system_call_after_swapgs+0xa2/0x13a [45777.306954] qmgr S ffff99ec6b5a5280 0 1955 1925 0x00000080 [45777.314908] Call Trace: [45777.317672] [] schedule+0x29/0x70 [45777.323228] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45777.331018] [] ? hrtimer_get_res+0x50/0x50 [45777.337459] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45777.345434] [] schedule_hrtimeout_range+0x13/0x20 [45777.352559] [] ep_poll+0x23e/0x360 [45777.358221] [] ? mntput+0x24/0x40 [45777.363786] [] ? wake_up_state+0x20/0x20 [45777.370036] [] SyS_epoll_wait+0xed/0x120 [45777.376274] [] ? system_call_after_swapgs+0x96/0x13a [45777.383673] [] system_call_fastpath+0x25/0x2a [45777.390401] [] ? system_call_after_swapgs+0xa2/0x13a [45777.397805] kworker/16:1H S ffff99e86b278000 0 1962 2 0x00000080 [45777.405745] Call Trace: [45777.408499] [] schedule+0x29/0x70 [45777.414065] [] worker_thread+0x1d9/0x3c0 [45777.420301] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45777.427617] [] kthread+0xd1/0xe0 [45777.433094] [] ? insert_kthread_work+0x40/0x40 [45777.439946] [] ret_from_fork_nospec_begin+0x21/0x21 [45777.447257] [] ? insert_kthread_work+0x40/0x40 [45777.454072] kworker/27:2 S ffff99ec5d1ae300 0 1989 2 0x00000080 [45777.462025] Call Trace: [45777.464792] [] schedule+0x29/0x70 [45777.470361] [] worker_thread+0x1d9/0x3c0 [45777.476604] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45777.483910] [] kthread+0xd1/0xe0 [45777.489381] [] ? insert_kthread_work+0x40/0x40 [45777.496207] [] ret_from_fork_nospec_begin+0x21/0x21 [45777.503519] [] ? insert_kthread_work+0x40/0x40 [45777.510336] agetty S ffff99e86c50b180 0 1994 1 0x00000080 [45777.518284] Call Trace: [45777.521039] [] schedule+0x29/0x70 [45777.526610] [] schedule_timeout+0x221/0x2d0 [45777.533145] [] ? flush_work+0x47/0x1b0 [45777.539196] [] ? serial8250_start_tx+0xa5/0xe0 [45777.546036] [] n_tty_read+0x2b0/0xd10 [45777.551988] [] ? task_rq_unlock+0x20/0x20 [45777.558348] [] ? wake_up_state+0x20/0x20 [45777.564591] [] tty_read+0x90/0x100 [45777.570254] [] vfs_read+0x9f/0x170 [45777.575917] [] SyS_read+0x7f/0xf0 [45777.581496] [] ? system_call_after_swapgs+0xa2/0x13a [45777.588906] [] system_call_fastpath+0x25/0x2a [45777.595621] [] ? system_call_after_swapgs+0xa2/0x13a [45777.603034] agetty S ffff99e86c50c200 0 1998 1 0x00000080 [45777.610992] Call Trace: [45777.613758] [] schedule+0x29/0x70 [45777.619315] [] schedule_timeout+0x221/0x2d0 [45777.625851] [] ? flush_work+0x47/0x1b0 [45777.631914] [] n_tty_read+0x2b0/0xd10 [45777.637866] [] ? task_rq_unlock+0x20/0x20 [45777.644206] [] ? wake_up_state+0x20/0x20 [45777.650435] [] tty_read+0x90/0x100 [45777.656096] [] vfs_read+0x9f/0x170 [45777.661759] [] SyS_read+0x7f/0xf0 [45777.667324] [] ? system_call_after_swapgs+0xa2/0x13a [45777.674731] [] system_call_fastpath+0x25/0x2a [45777.681461] [] ? system_call_after_swapgs+0xa2/0x13a [45777.688871] kworker/18:1H S ffff99e83c3ad280 0 2396 2 0x00000080 [45777.696846] Call Trace: [45777.699597] [] schedule+0x29/0x70 [45777.705156] [] worker_thread+0x1d9/0x3c0 [45777.711400] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45777.718710] [] kthread+0xd1/0xe0 [45777.724172] [] ? system_call_after_swapgs+0x96/0x13a [45777.731573] [] ? insert_kthread_work+0x40/0x40 [45777.738398] [] ret_from_fork_nospec_begin+0x21/0x21 [45777.745696] [] ? insert_kthread_work+0x40/0x40 [45777.752526] kworker/u308:1 S ffff99e86b27e300 0 2439 2 0x00000080 [45777.760460] Call Trace: [45777.763205] [] schedule+0x29/0x70 [45777.768776] [] worker_thread+0x1d9/0x3c0 [45777.775034] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45777.782345] [] kthread+0xd1/0xe0 [45777.787808] [] ? insert_kthread_work+0x40/0x40 [45777.794625] [] ret_from_fork_nospec_begin+0x21/0x21 [45777.801953] [] ? insert_kthread_work+0x40/0x40 [45777.808790] ntpd S ffff99ec6969c200 0 2693 1 0x00000080 [45777.816768] Call Trace: [45777.819529] [] schedule+0x29/0x70 [45777.825102] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45777.832906] [] ? hrtimer_get_res+0x50/0x50 [45777.839344] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45777.847334] [] schedule_hrtimeout_range+0x13/0x20 [45777.854452] [] poll_schedule_timeout+0x55/0xc0 [45777.861278] [] do_select+0x6d1/0x7c0 [45777.867142] [] ? put_cmsg+0xcd/0x150 [45777.872999] [] ? __pollwait+0xf0/0xf0 [45777.878971] [] ? __pollwait+0xf0/0xf0 [45777.884928] [] ? __pollwait+0xf0/0xf0 [45777.890897] [] ? __pollwait+0xf0/0xf0 [45777.896851] [] ? __pollwait+0xf0/0xf0 [45777.902805] [] ? __pollwait+0xf0/0xf0 [45777.908758] [] ? __pollwait+0xf0/0xf0 [45777.914710] [] ? __pollwait+0xf0/0xf0 [45777.920648] [] ? __pollwait+0xf0/0xf0 [45777.926600] [] core_sys_select+0x1ff/0x340 [45777.933049] [] ? __d_free+0x35/0x40 [45777.938799] [] ? d_free+0x60/0x70 [45777.944372] [] ? __dentry_kill+0x160/0x1d0 [45777.950840] [] ? ktime_get_ts64+0x52/0xf0 [45777.957180] [] SyS_select+0xca/0x120 [45777.963048] [] system_call_fastpath+0x25/0x2a [45777.969778] [] ? system_call_after_swapgs+0xa2/0x13a [45777.977190] kworker/u310:0 S ffff99ec6d474200 0 3447 2 0x00000080 [45777.985157] Call Trace: [45777.987916] [] schedule+0x29/0x70 [45777.993484] [] worker_thread+0x1d9/0x3c0 [45777.999729] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45778.007062] [] kthread+0xd1/0xe0 [45778.012531] [] ? insert_kthread_work+0x40/0x40 [45778.019357] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.026669] [] ? insert_kthread_work+0x40/0x40 [45778.033505] ptlrpc_hr00_000 S ffff99ec6acc5280 0 3451 2 0x00000080 [45778.041466] Call Trace: [45778.044221] [] schedule+0x29/0x70 [45778.049881] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.057092] [] ? wake_up_atomic_t+0x30/0x30 [45778.063700] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.071303] [] kthread+0xd1/0xe0 [45778.076779] [] ? insert_kthread_work+0x40/0x40 [45778.083605] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.090917] [] ? insert_kthread_work+0x40/0x40 [45778.097754] ptlrpc_hr00_001 S ffff99ec6acc2100 0 3453 2 0x00000080 [45778.105707] Call Trace: [45778.108459] [] schedule+0x29/0x70 [45778.114109] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.121331] [] ? wake_up_atomic_t+0x30/0x30 [45778.127924] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.135526] [] kthread+0xd1/0xe0 [45778.141013] [] ? insert_kthread_work+0x40/0x40 [45778.147857] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.155169] [] ? insert_kthread_work+0x40/0x40 [45778.161994] ptlrpc_hr00_002 S ffff99ec6acc6300 0 3454 2 0x00000080 [45778.169945] Call Trace: [45778.172697] [] schedule+0x29/0x70 [45778.178332] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.185557] [] ? wake_up_atomic_t+0x30/0x30 [45778.192161] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.199773] [] kthread+0xd1/0xe0 [45778.205256] [] ? insert_kthread_work+0x40/0x40 [45778.212083] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.219393] [] ? insert_kthread_work+0x40/0x40 [45778.226228] ptlrpc_hr00_003 S ffff99ec6acc0000 0 3455 2 0x00000080 [45778.234206] Call Trace: [45778.236981] [] schedule+0x29/0x70 [45778.242617] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.249845] [] ? wake_up_atomic_t+0x30/0x30 [45778.256450] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.264076] [] kthread+0xd1/0xe0 [45778.269559] [] ? insert_kthread_work+0x40/0x40 [45778.276387] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.283699] [] ? insert_kthread_work+0x40/0x40 [45778.290529] ptlrpc_hr00_004 S ffff99ec6acc1080 0 3456 2 0x00000080 [45778.298491] Call Trace: [45778.301236] [] schedule+0x29/0x70 [45778.306869] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.314087] [] ? wake_up_atomic_t+0x30/0x30 [45778.320676] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.328280] [] kthread+0xd1/0xe0 [45778.333763] [] ? insert_kthread_work+0x40/0x40 [45778.340583] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.347886] [] ? insert_kthread_work+0x40/0x40 [45778.354713] ptlrpc_hr00_005 S ffff99ec6d473180 0 3457 2 0x00000080 [45778.362681] Call Trace: [45778.365427] [] schedule+0x29/0x70 [45778.371077] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.378305] [] ? wake_up_atomic_t+0x30/0x30 [45778.384907] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.392519] [] kthread+0xd1/0xe0 [45778.398020] [] ? insert_kthread_work+0x40/0x40 [45778.404848] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.412160] [] ? insert_kthread_work+0x40/0x40 [45778.419013] ptlrpc_hr00_006 S ffff99ec6d472100 0 3458 2 0x00000080 [45778.426989] Call Trace: [45778.429751] [] schedule+0x29/0x70 [45778.435376] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.442601] [] ? wake_up_atomic_t+0x30/0x30 [45778.449204] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.456817] [] kthread+0xd1/0xe0 [45778.462299] [] ? insert_kthread_work+0x40/0x40 [45778.469128] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.476438] [] ? insert_kthread_work+0x40/0x40 [45778.483264] ptlrpc_hr00_007 S ffff99ec6d471080 0 3459 2 0x00000080 [45778.491215] Call Trace: [45778.493999] [] schedule+0x29/0x70 [45778.499624] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.506851] [] ? wake_up_atomic_t+0x30/0x30 [45778.513455] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.521059] [] kthread+0xd1/0xe0 [45778.526550] [] ? system_call_after_swapgs+0x96/0x13a [45778.533961] [] ? insert_kthread_work+0x40/0x40 [45778.540803] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.548109] [] ? insert_kthread_work+0x40/0x40 [45778.554943] ptlrpc_hr01_000 S ffff99ec6d476300 0 3460 2 0x00000080 [45778.562902] Call Trace: [45778.565664] [] schedule+0x29/0x70 [45778.571297] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.578529] [] ? wake_up_atomic_t+0x30/0x30 [45778.585149] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.592762] [] kthread+0xd1/0xe0 [45778.598239] [] ? insert_kthread_work+0x40/0x40 [45778.605080] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.612382] [] ? insert_kthread_work+0x40/0x40 [45778.619216] ptlrpc_hr01_001 S ffff99ec6d475280 0 3461 2 0x00000080 [45778.627133] Call Trace: [45778.629885] [] schedule+0x29/0x70 [45778.635500] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.642742] [] ? wake_up_atomic_t+0x30/0x30 [45778.649351] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.656956] [] kthread+0xd1/0xe0 [45778.662423] [] ? insert_kthread_work+0x40/0x40 [45778.669238] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.676542] [] ? insert_kthread_work+0x40/0x40 [45778.683381] ptlrpc_hr01_002 S ffff99ec6d470000 0 3462 2 0x00000080 [45778.691320] Call Trace: [45778.694073] [] schedule+0x29/0x70 [45778.699713] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.706947] [] ? wake_up_atomic_t+0x30/0x30 [45778.713534] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.721146] [] kthread+0xd1/0xe0 [45778.726622] [] ? insert_kthread_work+0x40/0x40 [45778.733459] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.740768] [] ? insert_kthread_work+0x40/0x40 [45778.747593] ptlrpc_hr01_003 S ffff99ec6df0c200 0 3463 2 0x00000080 [45778.755553] Call Trace: [45778.758299] [] schedule+0x29/0x70 [45778.763936] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.771156] [] ? wake_up_atomic_t+0x30/0x30 [45778.777769] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.785361] [] kthread+0xd1/0xe0 [45778.790837] [] ? insert_kthread_work+0x40/0x40 [45778.797666] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.804977] [] ? insert_kthread_work+0x40/0x40 [45778.811818] ptlrpc_hr01_004 S ffff99ec6df0b180 0 3464 2 0x00000080 [45778.819784] Call Trace: [45778.822518] [] schedule+0x29/0x70 [45778.828206] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.835434] [] ? wake_up_atomic_t+0x30/0x30 [45778.842046] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.849651] [] kthread+0xd1/0xe0 [45778.855128] [] ? insert_kthread_work+0x40/0x40 [45778.861955] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.869268] [] ? insert_kthread_work+0x40/0x40 [45778.876094] ptlrpc_hr01_005 S ffff99ec62ca0000 0 3465 2 0x00000080 [45778.884070] Call Trace: [45778.886835] [] schedule+0x29/0x70 [45778.892482] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.899715] [] ? wake_up_atomic_t+0x30/0x30 [45778.906324] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.913931] [] kthread+0xd1/0xe0 [45778.919408] [] ? insert_kthread_work+0x40/0x40 [45778.926234] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.933551] [] ? insert_kthread_work+0x40/0x40 [45778.940360] ptlrpc_hr01_006 S ffff99ec62ca1080 0 3466 2 0x00000080 [45778.948307] Call Trace: [45778.951099] [] schedule+0x29/0x70 [45778.956726] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45778.963946] [] ? wake_up_atomic_t+0x30/0x30 [45778.970540] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45778.978145] [] kthread+0xd1/0xe0 [45778.983625] [] ? insert_kthread_work+0x40/0x40 [45778.990457] [] ret_from_fork_nospec_begin+0x21/0x21 [45778.997769] [] ? insert_kthread_work+0x40/0x40 [45779.004593] ptlrpc_hr01_007 S ffff99ec62ca2100 0 3467 2 0x00000080 [45779.012554] Call Trace: [45779.015301] [] schedule+0x29/0x70 [45779.020928] [] ptlrpc_hr_main+0x49c/0x5b0 [ptlrpc] [45779.028146] [] ? wake_up_atomic_t+0x30/0x30 [45779.034744] [] ? ptlrpc_handle_rs+0x620/0x620 [ptlrpc] [45779.042348] [] kthread+0xd1/0xe0 [45779.047825] [] ? insert_kthread_work+0x40/0x40 [45779.054652] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.061966] [] ? insert_kthread_work+0x40/0x40 [45779.068790] kiblnd_connd S ffff99ec62ca3180 0 3477 2 0x00000080 [45779.076719] Call Trace: [45779.079465] [] schedule+0x29/0x70 [45779.085049] [] schedule_timeout+0x168/0x2d0 [45779.091585] [] ? requeue_timers+0x170/0x170 [45779.098138] [] kiblnd_connd+0x84d/0xcb0 [ko2iblnd] [45779.105339] [] ? wake_up_atomic_t+0x30/0x30 [45779.111889] [] ? kiblnd_cm_callback+0x2160/0x2160 [ko2iblnd] [45779.120086] [] kthread+0xd1/0xe0 [45779.125548] [] ? insert_kthread_work+0x40/0x40 [45779.132367] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.139669] [] ? insert_kthread_work+0x40/0x40 [45779.146500] kiblnd_sd_00_00 S ffff99ec62ca4200 0 3478 2 0x00000080 [45779.154453] Call Trace: [45779.157200] [] schedule+0x29/0x70 [45779.162777] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.170467] [] ? wake_up_atomic_t+0x30/0x30 [45779.177027] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.184530] [] kthread+0xd1/0xe0 [45779.190022] [] ? insert_kthread_work+0x40/0x40 [45779.196847] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.204150] [] ? insert_kthread_work+0x40/0x40 [45779.210977] kiblnd_sd_00_01 S ffff99ec6a5cd280 0 3479 2 0x00000080 [45779.218919] Call Trace: [45779.221664] [] schedule+0x29/0x70 [45779.227240] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.234947] [] ? wake_up_atomic_t+0x30/0x30 [45779.241507] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.249031] [] kthread+0xd1/0xe0 [45779.254506] [] ? insert_kthread_work+0x40/0x40 [45779.261332] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.268636] [] ? insert_kthread_work+0x40/0x40 [45779.275446] kiblnd_sd_00_02 S ffff99ec6a5ca100 0 3480 2 0x00000080 [45779.283394] Call Trace: [45779.286148] [] schedule+0x29/0x70 [45779.291718] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.299424] [] ? wake_up_atomic_t+0x30/0x30 [45779.305965] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.313470] [] kthread+0xd1/0xe0 [45779.318938] [] ? insert_kthread_work+0x40/0x40 [45779.325763] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.333073] [] ? insert_kthread_work+0x40/0x40 [45779.339902] kiblnd_sd_00_03 S ffff99ec6a5cc200 0 3481 2 0x00000080 [45779.347865] Call Trace: [45779.350601] [] schedule+0x29/0x70 [45779.356169] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.363875] [] ? wake_up_atomic_t+0x30/0x30 [45779.370403] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.377908] [] kthread+0xd1/0xe0 [45779.383364] [] ? insert_kthread_work+0x40/0x40 [45779.390183] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.397501] [] ? insert_kthread_work+0x40/0x40 [45779.404316] kiblnd_sd_01_00 S ffff99ec6a5c8000 0 3482 2 0x00000080 [45779.412265] Call Trace: [45779.415033] [] schedule+0x29/0x70 [45779.420590] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.428288] [] ? wake_up_atomic_t+0x30/0x30 [45779.434838] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.442342] [] kthread+0xd1/0xe0 [45779.447810] [] ? insert_kthread_work+0x40/0x40 [45779.454643] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.461957] [] ? insert_kthread_work+0x40/0x40 [45779.468780] kiblnd_sd_01_01 S ffff99ec6a5cb180 0 3483 2 0x00000080 [45779.476723] Call Trace: [45779.479484] [] schedule+0x29/0x70 [45779.485072] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.492783] [] ? wake_up_atomic_t+0x30/0x30 [45779.499315] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.506827] [] kthread+0xd1/0xe0 [45779.512301] [] ? insert_kthread_work+0x40/0x40 [45779.519119] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.526430] [] ? insert_kthread_work+0x40/0x40 [45779.533254] kiblnd_sd_01_02 S ffff99ec6a5ce300 0 3484 2 0x00000080 [45779.541208] Call Trace: [45779.543945] [] schedule+0x29/0x70 [45779.549523] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.557226] [] ? wake_up_atomic_t+0x30/0x30 [45779.563767] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.571272] [] kthread+0xd1/0xe0 [45779.576747] [] ? insert_kthread_work+0x40/0x40 [45779.583571] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.590873] [] ? insert_kthread_work+0x40/0x40 [45779.597695] kiblnd_sd_01_03 S ffff99ec6a5c9080 0 3485 2 0x00000080 [45779.605663] Call Trace: [45779.608432] [] schedule+0x29/0x70 [45779.614014] [] kiblnd_scheduler+0x5fe/0x11e0 [ko2iblnd] [45779.621748] [] ? wake_up_atomic_t+0x30/0x30 [45779.628290] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [45779.635788] [] kthread+0xd1/0xe0 [45779.641258] [] ? insert_kthread_work+0x40/0x40 [45779.648083] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.655397] [] ? insert_kthread_work+0x40/0x40 [45779.662221] ib_fmr(mlx4_0) S ffff99e86b148000 0 3486 2 0x00000080 [45779.670168] Call Trace: [45779.672919] [] schedule+0x29/0x70 [45779.678535] [] ib_fmr_cleanup_thread+0x7b/0xd0 [ib_core] [45779.686344] [] ? ib_fmr_batch_release+0x180/0x180 [ib_core] [45779.694427] [] kthread+0xd1/0xe0 [45779.699896] [] ? insert_kthread_work+0x40/0x40 [45779.706721] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.714050] [] ? insert_kthread_work+0x40/0x40 [45779.720882] ib_fmr(mlx4_0) S ffff99e86a4e2100 0 3487 2 0x00000080 [45779.728857] Call Trace: [45779.731625] [] schedule+0x29/0x70 [45779.737207] [] ib_fmr_cleanup_thread+0x7b/0xd0 [ib_core] [45779.745045] [] ? ib_fmr_batch_release+0x180/0x180 [ib_core] [45779.753131] [] kthread+0xd1/0xe0 [45779.758605] [] ? insert_kthread_work+0x40/0x40 [45779.765418] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.772721] [] ? insert_kthread_work+0x40/0x40 [45779.779546] lnet_discovery S ffff99e86a4e4200 0 3488 2 0x00000080 [45779.787497] Call Trace: [45779.790236] [] schedule+0x29/0x70 [45779.795802] [] schedule_timeout+0x168/0x2d0 [45779.802343] [] ? requeue_timers+0x170/0x170 [45779.808925] [] lnet_peer_discovery+0xe9/0x1600 [lnet] [45779.816437] [] ? __schedule+0x320/0x680 [45779.822585] [] ? wake_up_atomic_t+0x30/0x30 [45779.829153] [] ? lnet_peer_merge_data+0x1120/0x1120 [lnet] [45779.837184] [] kthread+0xd1/0xe0 [45779.842662] [] ? insert_kthread_work+0x40/0x40 [45779.849491] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.856802] [] ? insert_kthread_work+0x40/0x40 [45779.863639] monitor_thread S ffff99e86a4e6300 0 3489 2 0x00000080 [45779.871604] Call Trace: [45779.874350] [] schedule+0x29/0x70 [45779.879914] [] schedule_timeout+0x168/0x2d0 [45779.886450] [] ? requeue_timers+0x170/0x170 [45779.892990] [] wait_for_completion_interruptible_timeout+0x111/0x170 [45779.901959] [] ? wake_up_state+0x20/0x20 [45779.908214] [] lnet_monitor_thread+0x1aa/0x440 [lnet] [45779.915726] [] ? lnet_recover_peer_nis+0x6a0/0x6a0 [lnet] [45779.923624] [] kthread+0xd1/0xe0 [45779.929099] [] ? insert_kthread_work+0x40/0x40 [45779.935944] [] ret_from_fork_nospec_begin+0x21/0x21 [45779.943254] [] ? insert_kthread_work+0x40/0x40 [45779.950068] ptlrpcd_rcv S ffff99e86a4e3180 0 3490 2 0x00000080 [45779.958043] Call Trace: [45779.960796] [] schedule+0x29/0x70 [45779.966360] [] schedule_timeout+0x221/0x2d0 [45779.972984] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45779.980584] [] ? kmem_cache_free+0x1e2/0x200 [45779.987297] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45779.995693] [] wait_woken+0x66/0x90 [45780.001534] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.008084] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.015090] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.022695] [] kthread+0xd1/0xe0 [45780.028149] [] ? insert_kthread_work+0x40/0x40 [45780.034975] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.042287] [] ? insert_kthread_work+0x40/0x40 [45780.049122] ptlrpcd_00_00 S ffff99e86a4e0000 0 3491 2 0x00000080 [45780.057106] Call Trace: [45780.059866] [] schedule+0x29/0x70 [45780.065416] [] schedule_timeout+0x221/0x2d0 [45780.072037] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.079642] [] ? kmem_cache_free+0x1e2/0x200 [45780.086345] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.094712] [] wait_woken+0x66/0x90 [45780.100536] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.107098] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.114111] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.121716] [] kthread+0xd1/0xe0 [45780.127198] [] ? insert_kthread_work+0x40/0x40 [45780.134028] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.141338] [] ? insert_kthread_work+0x40/0x40 [45780.148157] ptlrpcd_00_01 S ffff99e86a4e1080 0 3492 2 0x00000080 [45780.156100] Call Trace: [45780.158854] [] schedule+0x29/0x70 [45780.164413] [] schedule_timeout+0x221/0x2d0 [45780.171012] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.178626] [] ? kmem_cache_free+0x1e2/0x200 [45780.185313] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.193687] [] wait_woken+0x66/0x90 [45780.199521] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.206082] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.213086] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.220683] [] kthread+0xd1/0xe0 [45780.226164] [] ? insert_kthread_work+0x40/0x40 [45780.232990] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.240293] [] ? insert_kthread_work+0x40/0x40 [45780.247141] ptlrpcd_00_02 S ffff99ec6c9e8000 0 3493 2 0x00000080 [45780.255103] Call Trace: [45780.257856] [] schedule+0x29/0x70 [45780.263421] [] schedule_timeout+0x221/0x2d0 [45780.270020] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.277641] [] ? kmem_cache_free+0x1e2/0x200 [45780.284335] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.292715] [] wait_woken+0x66/0x90 [45780.298547] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.305098] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.312109] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.319723] [] kthread+0xd1/0xe0 [45780.325178] [] ? system_call_after_swapgs+0x96/0x13a [45780.332584] [] ? insert_kthread_work+0x40/0x40 [45780.339409] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.346721] [] ? insert_kthread_work+0x40/0x40 [45780.353546] ptlrpcd_00_03 S ffff99ec6c9e9080 0 3494 2 0x00000080 [45780.361490] Call Trace: [45780.364243] [] schedule+0x29/0x70 [45780.369808] [] schedule_timeout+0x221/0x2d0 [45780.376424] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.384022] [] ? kmem_cache_free+0x1e2/0x200 [45780.390721] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.399115] [] wait_woken+0x66/0x90 [45780.404941] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.411482] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.418475] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.426095] [] kthread+0xd1/0xe0 [45780.431570] [] ? insert_kthread_work+0x40/0x40 [45780.438403] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.445722] [] ? insert_kthread_work+0x40/0x40 [45780.452546] ptlrpcd_00_04 S ffff99ec6c9ea100 0 3495 2 0x00000080 [45780.460498] Call Trace: [45780.463260] [] schedule+0x29/0x70 [45780.468817] [] schedule_timeout+0x221/0x2d0 [45780.475412] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.483021] [] ? kmem_cache_free+0x1e2/0x200 [45780.489720] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.498115] [] wait_woken+0x66/0x90 [45780.503961] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.510508] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.517503] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.525109] [] kthread+0xd1/0xe0 [45780.530582] [] ? insert_kthread_work+0x40/0x40 [45780.537420] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.544732] [] ? insert_kthread_work+0x40/0x40 [45780.551557] ptlrpcd_00_05 S ffff99ec6c9eb180 0 3496 2 0x00000080 [45780.559501] Call Trace: [45780.562244] [] schedule+0x29/0x70 [45780.567824] [] schedule_timeout+0x221/0x2d0 [45780.574441] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.582050] [] ? kmem_cache_free+0x1e2/0x200 [45780.588750] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.597145] [] wait_woken+0x66/0x90 [45780.602983] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.609523] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.616516] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.624121] [] kthread+0xd1/0xe0 [45780.629600] [] ? insert_kthread_work+0x40/0x40 [45780.636425] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.643737] [] ? insert_kthread_work+0x40/0x40 [45780.650562] ptlrpcd_00_06 S ffff99ec6c9ec200 0 3497 2 0x00000080 [45780.658514] Call Trace: [45780.661269] [] schedule+0x29/0x70 [45780.666840] [] schedule_timeout+0x221/0x2d0 [45780.673448] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.681056] [] ? kmem_cache_free+0x1e2/0x200 [45780.687754] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.696162] [] wait_woken+0x66/0x90 [45780.702001] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.708540] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.715537] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.723155] [] kthread+0xd1/0xe0 [45780.728640] [] ? insert_kthread_work+0x40/0x40 [45780.735468] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.742780] [] ? insert_kthread_work+0x40/0x40 [45780.749602] ptlrpcd_00_07 S ffff99ec6c9ed280 0 3498 2 0x00000080 [45780.757569] Call Trace: [45780.760317] [] schedule+0x29/0x70 [45780.765890] [] schedule_timeout+0x221/0x2d0 [45780.772492] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.780120] [] ? kmem_cache_free+0x1e2/0x200 [45780.786820] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.795188] [] wait_woken+0x66/0x90 [45780.801026] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.807566] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.814560] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.822189] [] kthread+0xd1/0xe0 [45780.827671] [] ? system_call_after_swapgs+0x96/0x13a [45780.835164] [] ? insert_kthread_work+0x40/0x40 [45780.841993] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.849297] [] ? insert_kthread_work+0x40/0x40 [45780.856127] ptlrpcd_00_08 S ffff99ec6c9ee300 0 3499 2 0x00000080 [45780.864099] Call Trace: [45780.866861] [] schedule+0x29/0x70 [45780.872425] [] schedule_timeout+0x221/0x2d0 [45780.879036] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.886657] [] ? kmem_cache_free+0x1e2/0x200 [45780.893364] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45780.901753] [] wait_woken+0x66/0x90 [45780.907583] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45780.914129] [] ? abort_exclusive_wait+0xa0/0xa0 [45780.921134] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45780.928741] [] kthread+0xd1/0xe0 [45780.934211] [] ? insert_kthread_work+0x40/0x40 [45780.941042] [] ret_from_fork_nospec_begin+0x21/0x21 [45780.948353] [] ? insert_kthread_work+0x40/0x40 [45780.955174] ptlrpcd_00_09 S ffff99ec5d2c0000 0 3500 2 0x00000080 [45780.963121] Call Trace: [45780.965872] [] schedule+0x29/0x70 [45780.971437] [] schedule_timeout+0x221/0x2d0 [45780.978039] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45780.985647] [] ? kmem_cache_free+0x1e2/0x200 [45780.992347] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.000736] [] wait_woken+0x66/0x90 [45781.006568] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.013136] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.020134] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.027741] [] kthread+0xd1/0xe0 [45781.033209] [] ? insert_kthread_work+0x40/0x40 [45781.040030] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.047340] [] ? insert_kthread_work+0x40/0x40 [45781.054176] ptlrpcd_00_10 S ffff99ec5d2c1080 0 3501 2 0x00000080 [45781.062126] Call Trace: [45781.064887] [] schedule+0x29/0x70 [45781.070452] [] schedule_timeout+0x221/0x2d0 [45781.077064] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.084684] [] ? kmem_cache_free+0x1e2/0x200 [45781.091387] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.099775] [] wait_woken+0x66/0x90 [45781.105610] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.112149] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.119164] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.126769] [] kthread+0xd1/0xe0 [45781.132239] [] ? insert_kthread_work+0x40/0x40 [45781.139068] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.146386] [] ? insert_kthread_work+0x40/0x40 [45781.153219] ptlrpcd_00_11 S ffff99ec5d2c2100 0 3502 2 0x00000080 [45781.161181] Call Trace: [45781.163925] [] schedule+0x29/0x70 [45781.169490] [] schedule_timeout+0x221/0x2d0 [45781.176116] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.183723] [] ? kmem_cache_free+0x1e2/0x200 [45781.190422] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.198809] [] wait_woken+0x66/0x90 [45781.204640] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.211198] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.218208] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.225813] [] kthread+0xd1/0xe0 [45781.231289] [] ? insert_kthread_work+0x40/0x40 [45781.238138] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.245450] [] ? insert_kthread_work+0x40/0x40 [45781.252267] ptlrpcd_00_12 S ffff99ec5d2c3180 0 3503 2 0x00000080 [45781.260200] Call Trace: [45781.262961] [] schedule+0x29/0x70 [45781.268527] [] schedule_timeout+0x221/0x2d0 [45781.275132] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.282736] [] ? kmem_cache_free+0x1e2/0x200 [45781.289437] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.297820] [] wait_woken+0x66/0x90 [45781.303650] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.310189] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.317194] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.324791] [] kthread+0xd1/0xe0 [45781.330253] [] ? insert_kthread_work+0x40/0x40 [45781.337083] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.344381] [] ? insert_kthread_work+0x40/0x40 [45781.351207] ptlrpcd_00_13 S ffff99ec5d2c4200 0 3504 2 0x00000080 [45781.359159] Call Trace: [45781.361931] [] schedule+0x29/0x70 [45781.367499] [] schedule_timeout+0x221/0x2d0 [45781.374127] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.381761] [] ? kmem_cache_free+0x1e2/0x200 [45781.388456] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.396846] [] wait_woken+0x66/0x90 [45781.402683] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.409230] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.416223] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.423843] [] kthread+0xd1/0xe0 [45781.429319] [] ? insert_kthread_work+0x40/0x40 [45781.436139] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.443450] [] ? insert_kthread_work+0x40/0x40 [45781.450273] ptlrpcd_00_14 S ffff99ec5d2c5280 0 3505 2 0x00000080 [45781.458229] Call Trace: [45781.460990] [] schedule+0x29/0x70 [45781.466560] [] schedule_timeout+0x221/0x2d0 [45781.473202] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.480799] [] ? kmem_cache_free+0x1e2/0x200 [45781.487499] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.495889] [] wait_woken+0x66/0x90 [45781.501723] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.508256] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.515255] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.522868] [] kthread+0xd1/0xe0 [45781.528355] [] ? insert_kthread_work+0x40/0x40 [45781.535197] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.542508] [] ? insert_kthread_work+0x40/0x40 [45781.549333] ptlrpcd_00_15 S ffff99ec5d2c6300 0 3506 2 0x00000080 [45781.557269] Call Trace: [45781.560017] [] schedule+0x29/0x70 [45781.565573] [] schedule_timeout+0x221/0x2d0 [45781.572201] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.579800] [] ? kmem_cache_free+0x1e2/0x200 [45781.586506] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.594888] [] wait_woken+0x66/0x90 [45781.600727] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.607268] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.614274] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.621881] [] kthread+0xd1/0xe0 [45781.627359] [] ? insert_kthread_work+0x40/0x40 [45781.634200] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.641511] [] ? insert_kthread_work+0x40/0x40 [45781.648342] ptlrpcd_01_00 S ffff99ec62c50000 0 3507 2 0x00000080 [45781.656310] Call Trace: [45781.659088] [] schedule+0x29/0x70 [45781.664668] [] schedule_timeout+0x221/0x2d0 [45781.671297] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.678903] [] ? kmem_cache_free+0x1e2/0x200 [45781.685605] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.693994] [] wait_woken+0x66/0x90 [45781.699825] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.706373] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.713368] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.720973] [] kthread+0xd1/0xe0 [45781.726456] [] ? insert_kthread_work+0x40/0x40 [45781.733284] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.740595] [] ? insert_kthread_work+0x40/0x40 [45781.747425] ptlrpcd_01_01 S ffff99ec62c51080 0 3508 2 0x00000080 [45781.755368] Call Trace: [45781.758145] [] schedule+0x29/0x70 [45781.763709] [] schedule_timeout+0x221/0x2d0 [45781.770319] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.777926] [] ? kmem_cache_free+0x1e2/0x200 [45781.784632] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.793028] [] wait_woken+0x66/0x90 [45781.798860] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.805406] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.812392] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.819984] [] kthread+0xd1/0xe0 [45781.825455] [] ? insert_kthread_work+0x40/0x40 [45781.832281] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.839591] [] ? insert_kthread_work+0x40/0x40 [45781.846431] ptlrpcd_01_02 S ffff99ec62c52100 0 3509 2 0x00000080 [45781.854405] Call Trace: [45781.857165] [] schedule+0x29/0x70 [45781.862731] [] schedule_timeout+0x221/0x2d0 [45781.869341] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.876946] [] ? kmem_cache_free+0x1e2/0x200 [45781.883649] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.892030] [] wait_woken+0x66/0x90 [45781.897858] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45781.904405] [] ? abort_exclusive_wait+0xa0/0xa0 [45781.911407] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45781.919019] [] kthread+0xd1/0xe0 [45781.924488] [] ? insert_kthread_work+0x40/0x40 [45781.931316] [] ret_from_fork_nospec_begin+0x21/0x21 [45781.938628] [] ? insert_kthread_work+0x40/0x40 [45781.945466] ptlrpcd_01_03 S ffff99ec62c53180 0 3510 2 0x00000080 [45781.953403] Call Trace: [45781.956177] [] schedule+0x29/0x70 [45781.961740] [] schedule_timeout+0x221/0x2d0 [45781.968358] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45781.975972] [] ? kmem_cache_free+0x1e2/0x200 [45781.982671] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45781.991061] [] wait_woken+0x66/0x90 [45781.996883] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.003430] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.010423] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.018024] [] kthread+0xd1/0xe0 [45782.023493] [] ? insert_kthread_work+0x40/0x40 [45782.030319] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.037636] [] ? insert_kthread_work+0x40/0x40 [45782.044468] ptlrpcd_01_04 S ffff99ec62c54200 0 3511 2 0x00000080 [45782.052433] Call Trace: [45782.055206] [] schedule+0x29/0x70 [45782.060771] [] schedule_timeout+0x221/0x2d0 [45782.067388] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.074986] [] ? kmem_cache_free+0x1e2/0x200 [45782.081686] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.090073] [] wait_woken+0x66/0x90 [45782.095910] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.102465] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.109466] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.117073] [] kthread+0xd1/0xe0 [45782.122544] [] ? insert_kthread_work+0x40/0x40 [45782.129372] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.136684] [] ? insert_kthread_work+0x40/0x40 [45782.143520] ptlrpcd_01_05 S ffff99ec62c55280 0 3512 2 0x00000080 [45782.151458] Call Trace: [45782.154223] [] schedule+0x29/0x70 [45782.159782] [] schedule_timeout+0x221/0x2d0 [45782.166408] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.174021] [] ? kmem_cache_free+0x1e2/0x200 [45782.180721] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.189102] [] wait_woken+0x66/0x90 [45782.194921] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.201467] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.208461] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.216065] [] kthread+0xd1/0xe0 [45782.221533] [] ? insert_kthread_work+0x40/0x40 [45782.228361] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.235678] [] ? insert_kthread_work+0x40/0x40 [45782.242507] ptlrpcd_01_06 S ffff99ec62c56300 0 3513 2 0x00000080 [45782.250466] Call Trace: [45782.253237] [] schedule+0x29/0x70 [45782.258810] [] schedule_timeout+0x221/0x2d0 [45782.265426] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.273023] [] ? kmem_cache_free+0x1e2/0x200 [45782.279727] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.288110] [] wait_woken+0x66/0x90 [45782.293950] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.300496] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.307490] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.315089] [] kthread+0xd1/0xe0 [45782.320559] [] ? insert_kthread_work+0x40/0x40 [45782.327388] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.334698] [] ? insert_kthread_work+0x40/0x40 [45782.341524] ptlrpcd_01_07 S ffff99ec5c1c0000 0 3514 2 0x00000080 [45782.349503] Call Trace: [45782.352251] [] schedule+0x29/0x70 [45782.357821] [] schedule_timeout+0x221/0x2d0 [45782.364424] [] ? ptlrpc_check_set+0x18ec/0x2290 [ptlrpc] [45782.372239] [] ? requeue_timers+0x170/0x170 [45782.378805] [] wait_woken+0x66/0x90 [45782.384637] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.391184] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.398191] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.405798] [] kthread+0xd1/0xe0 [45782.411271] [] ? insert_kthread_work+0x40/0x40 [45782.418095] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.425414] [] ? insert_kthread_work+0x40/0x40 [45782.432257] ptlrpcd_01_08 S ffff99ec5c1c1080 0 3515 2 0x00000080 [45782.440238] Call Trace: [45782.442992] [] schedule+0x29/0x70 [45782.448556] [] schedule_timeout+0x221/0x2d0 [45782.455173] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.462786] [] ? kmem_cache_free+0x1e2/0x200 [45782.469486] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.477867] [] wait_woken+0x66/0x90 [45782.483706] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.490263] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.497263] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.504861] [] kthread+0xd1/0xe0 [45782.510330] [] ? insert_kthread_work+0x40/0x40 [45782.517178] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.524483] [] ? insert_kthread_work+0x40/0x40 [45782.531312] ptlrpcd_01_09 S ffff99ec5c1c2100 0 3516 2 0x00000080 [45782.539257] Call Trace: [45782.542012] [] schedule+0x29/0x70 [45782.547583] [] schedule_timeout+0x221/0x2d0 [45782.554212] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.561825] [] ? kmem_cache_free+0x1e2/0x200 [45782.568523] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.576906] [] wait_woken+0x66/0x90 [45782.582739] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.589277] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.596261] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.603867] [] kthread+0xd1/0xe0 [45782.609325] [] ? insert_kthread_work+0x40/0x40 [45782.616174] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.623471] [] ? insert_kthread_work+0x40/0x40 [45782.630288] ptlrpcd_01_10 S ffff99ec5c1c3180 0 3517 2 0x00000080 [45782.638210] Call Trace: [45782.640962] [] schedule+0x29/0x70 [45782.646521] [] schedule_timeout+0x221/0x2d0 [45782.653125] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.660717] [] ? kmem_cache_free+0x1e2/0x200 [45782.667416] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.675808] [] wait_woken+0x66/0x90 [45782.681662] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.688210] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.695194] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.702807] [] kthread+0xd1/0xe0 [45782.708272] [] ? insert_kthread_work+0x40/0x40 [45782.715095] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.722404] [] ? insert_kthread_work+0x40/0x40 [45782.729240] ptlrpcd_01_11 S ffff99ec5c1c4200 0 3518 2 0x00000080 [45782.737201] Call Trace: [45782.739947] [] schedule+0x29/0x70 [45782.745517] [] schedule_timeout+0x221/0x2d0 [45782.752122] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.759724] [] ? kmem_cache_free+0x1e2/0x200 [45782.766423] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.774797] [] wait_woken+0x66/0x90 [45782.780622] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.787185] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.794193] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.801800] [] kthread+0xd1/0xe0 [45782.807279] [] ? insert_kthread_work+0x40/0x40 [45782.814113] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.821423] [] ? insert_kthread_work+0x40/0x40 [45782.828259] ptlrpcd_01_12 S ffff99ec5c1c5280 0 3519 2 0x00000080 [45782.836221] Call Trace: [45782.838975] [] schedule+0x29/0x70 [45782.844534] [] schedule_timeout+0x221/0x2d0 [45782.851140] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.858755] [] ? kmem_cache_free+0x1e2/0x200 [45782.865441] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.873829] [] wait_woken+0x66/0x90 [45782.879669] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.886217] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.893211] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.900810] [] kthread+0xd1/0xe0 [45782.906279] [] ? insert_kthread_work+0x40/0x40 [45782.913108] [] ret_from_fork_nospec_begin+0x21/0x21 [45782.920419] [] ? insert_kthread_work+0x40/0x40 [45782.927254] ptlrpcd_01_13 S ffff99ec5c1c6300 0 3520 2 0x00000080 [45782.935226] Call Trace: [45782.937998] [] schedule+0x29/0x70 [45782.943571] [] schedule_timeout+0x221/0x2d0 [45782.950195] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45782.957809] [] ? kmem_cache_free+0x1e2/0x200 [45782.964504] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45782.972877] [] wait_woken+0x66/0x90 [45782.978709] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45782.985252] [] ? abort_exclusive_wait+0xa0/0xa0 [45782.992264] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45782.999862] [] kthread+0xd1/0xe0 [45783.005338] [] ? insert_kthread_work+0x40/0x40 [45783.012166] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.019490] [] ? insert_kthread_work+0x40/0x40 [45783.026322] ptlrpcd_01_14 S ffff99ec5c1f0000 0 3521 2 0x00000080 [45783.034276] Call Trace: [45783.037035] [] schedule+0x29/0x70 [45783.042601] [] schedule_timeout+0x221/0x2d0 [45783.049221] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45783.056825] [] ? kmem_cache_free+0x1e2/0x200 [45783.063535] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45783.071925] [] wait_woken+0x66/0x90 [45783.077757] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45783.084305] [] ? abort_exclusive_wait+0xa0/0xa0 [45783.091310] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45783.098922] [] kthread+0xd1/0xe0 [45783.104392] [] ? insert_kthread_work+0x40/0x40 [45783.111226] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.118539] [] ? insert_kthread_work+0x40/0x40 [45783.125370] ptlrpcd_01_15 S ffff99ec5c1f1080 0 3522 2 0x00000080 [45783.133328] Call Trace: [45783.136081] [] schedule+0x29/0x70 [45783.141645] [] schedule_timeout+0x221/0x2d0 [45783.148271] [] ? null_free_reqbuf+0x134/0x2f0 [ptlrpc] [45783.155878] [] ? kmem_cache_free+0x1e2/0x200 [45783.162570] [] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] [45783.170959] [] wait_woken+0x66/0x90 [45783.176791] [] ptlrpcd+0x99e/0xb00 [ptlrpc] [45783.183320] [] ? abort_exclusive_wait+0xa0/0xa0 [45783.190335] [] ? ptlrpcd_partners+0x380/0x380 [ptlrpc] [45783.197949] [] kthread+0xd1/0xe0 [45783.203420] [] ? insert_kthread_work+0x40/0x40 [45783.210266] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.217575] [] ? insert_kthread_work+0x40/0x40 [45783.224401] kworker/6:2 S ffff99e824239080 0 3528 2 0x00000080 [45783.232334] Call Trace: [45783.235082] [] schedule+0x29/0x70 [45783.240647] [] worker_thread+0x1d9/0x3c0 [45783.246890] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45783.254219] [] kthread+0xd1/0xe0 [45783.259681] [] ? insert_kthread_work+0x40/0x40 [45783.266505] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.273824] [] ? insert_kthread_work+0x40/0x40 [45783.280648] kmmpd-dm-3 S ffff99e82423a100 0 3531 2 0x00000080 [45783.288574] Call Trace: [45783.291322] [] schedule+0x29/0x70 [45783.296885] [] schedule_timeout+0x168/0x2d0 [45783.303421] [] ? requeue_timers+0x170/0x170 [45783.309988] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [45783.317610] [] schedule_timeout_interruptible+0x1e/0x20 [45783.325331] [] kmmpd+0x216/0x430 [ldiskfs] [45783.331776] [] ? __schedule+0x320/0x680 [45783.337933] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [45783.345268] [] kthread+0xd1/0xe0 [45783.350770] [] ? system_call_after_swapgs+0x96/0x13a [45783.358199] [] ? insert_kthread_work+0x40/0x40 [45783.365025] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.372315] [] ? insert_kthread_work+0x40/0x40 [45783.379175] jbd2/dm-3-8 S ffff99e82423b180 0 3532 2 0x00000080 [45783.387141] Call Trace: [45783.389883] [] schedule+0x29/0x70 [45783.395466] [] kjournald2+0x203/0x260 [jbd2] [45783.402110] [] ? wake_up_atomic_t+0x30/0x30 [45783.408642] [] ? commit_timeout+0x10/0x10 [jbd2] [45783.415662] [] kthread+0xd1/0xe0 [45783.421129] [] ? insert_kthread_work+0x40/0x40 [45783.427948] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.435257] [] ? insert_kthread_work+0x40/0x40 [45783.442098] ldiskfs-rsv-con S ffff99e82423c200 0 3533 2 0x00000080 [45783.450095] Call Trace: [45783.452863] [] schedule+0x29/0x70 [45783.458434] [] rescuer_thread+0x31e/0x3c0 [45783.464761] [] ? __schedule+0x320/0x680 [45783.470907] [] ? process_one_work+0x440/0x440 [45783.477637] [] kthread+0xd1/0xe0 [45783.483109] [] ? insert_kthread_work+0x40/0x40 [45783.489921] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.497230] [] ? insert_kthread_work+0x40/0x40 [45783.504054] ldlm_cb00_000 S ffff99e82423d280 0 3538 2 0x00000080 [45783.512045] Call Trace: [45783.514791] [] schedule+0x29/0x70 [45783.520432] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.527948] [] ? wake_up_atomic_t+0x30/0x30 [45783.534558] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.541671] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.549375] [] kthread+0xd1/0xe0 [45783.554844] [] ? insert_kthread_work+0x40/0x40 [45783.561673] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.568984] [] ? insert_kthread_work+0x40/0x40 [45783.575816] ldlm_cb00_001 S ffff99e82423e300 0 3539 2 0x00000080 [45783.583796] Call Trace: [45783.586551] [] schedule+0x29/0x70 [45783.592175] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.599692] [] ? wake_up_atomic_t+0x30/0x30 [45783.606311] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.613417] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.621111] [] kthread+0xd1/0xe0 [45783.626587] [] ? system_call_after_swapgs+0x96/0x13a [45783.633993] [] ? insert_kthread_work+0x40/0x40 [45783.640828] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.648150] [] ? insert_kthread_work+0x40/0x40 [45783.654981] ldlm_cb01_000 S ffff99ec5c1f6300 0 3540 2 0x00000080 [45783.662940] Call Trace: [45783.665702] [] schedule+0x29/0x70 [45783.671349] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.678867] [] ? wake_up_atomic_t+0x30/0x30 [45783.685475] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.692565] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.700251] [] kthread+0xd1/0xe0 [45783.705727] [] ? insert_kthread_work+0x40/0x40 [45783.712561] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.719868] [] ? insert_kthread_work+0x40/0x40 [45783.726707] ldlm_cb01_001 S ffff99ec5c1f5280 0 3541 2 0x00000080 [45783.734654] Call Trace: [45783.737408] [] schedule+0x29/0x70 [45783.743041] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.750556] [] ? wake_up_atomic_t+0x30/0x30 [45783.757156] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.764268] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.771982] [] kthread+0xd1/0xe0 [45783.777464] [] ? insert_kthread_work+0x40/0x40 [45783.784302] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.791613] [] ? insert_kthread_work+0x40/0x40 [45783.798445] ldlm_cn00_000 S ffff99ec5c1f4200 0 3542 2 0x00000080 [45783.806388] Call Trace: [45783.809141] [] schedule+0x29/0x70 [45783.814749] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.822268] [] ? wake_up_atomic_t+0x30/0x30 [45783.828887] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.836004] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.843706] [] kthread+0xd1/0xe0 [45783.849171] [] ? insert_kthread_work+0x40/0x40 [45783.856000] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.863315] [] ? insert_kthread_work+0x40/0x40 [45783.870162] ldlm_cn00_001 S ffff99ec5c1f3180 0 3543 2 0x00000080 [45783.878132] Call Trace: [45783.880895] [] schedule+0x29/0x70 [45783.886531] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.894074] [] ? wake_up_atomic_t+0x30/0x30 [45783.900678] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.907782] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.915484] [] kthread+0xd1/0xe0 [45783.920953] [] ? insert_kthread_work+0x40/0x40 [45783.927779] [] ret_from_fork_nospec_begin+0x21/0x21 [45783.935098] [] ? insert_kthread_work+0x40/0x40 [45783.941928] ldlm_cn00_002 S ffff99ec5c1f2100 0 3544 2 0x00000080 [45783.949886] Call Trace: [45783.952647] [] schedule+0x29/0x70 [45783.958289] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45783.965819] [] ? wake_up_atomic_t+0x30/0x30 [45783.972419] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45783.979526] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45783.987249] [] kthread+0xd1/0xe0 [45783.992718] [] ? insert_kthread_work+0x40/0x40 [45783.999547] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.006841] [] ? insert_kthread_work+0x40/0x40 [45784.013686] ldlm_cn01_000 S ffff99ec5c3b8000 0 3545 2 0x00000080 [45784.021659] Call Trace: [45784.024418] [] schedule+0x29/0x70 [45784.030057] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.037567] [] ? wake_up_atomic_t+0x30/0x30 [45784.044161] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.051270] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.058974] [] kthread+0xd1/0xe0 [45784.064442] [] ? insert_kthread_work+0x40/0x40 [45784.071269] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.078580] [] ? insert_kthread_work+0x40/0x40 [45784.085412] ldlm_cn01_001 S ffff99ec5c3b9080 0 3546 2 0x00000080 [45784.093358] Call Trace: [45784.096114] [] schedule+0x29/0x70 [45784.101726] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.109255] [] ? wake_up_atomic_t+0x30/0x30 [45784.115866] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.122967] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.130661] [] kthread+0xd1/0xe0 [45784.136121] [] ? insert_kthread_work+0x40/0x40 [45784.142949] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.150275] [] ? insert_kthread_work+0x40/0x40 [45784.157106] ldlm_cn01_002 S ffff99ec5c3ba100 0 3547 2 0x00000080 [45784.165089] Call Trace: [45784.167854] [] schedule+0x29/0x70 [45784.173488] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.180999] [] ? wake_up_atomic_t+0x30/0x30 [45784.187593] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.194689] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.202392] [] kthread+0xd1/0xe0 [45784.207861] [] ? insert_kthread_work+0x40/0x40 [45784.214687] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.222014] [] ? insert_kthread_work+0x40/0x40 [45784.228852] ldlm_bl_01 S ffff99ec5c3bb180 0 3548 2 0x00000080 [45784.236817] Call Trace: [45784.239573] [] schedule+0x29/0x70 [45784.245184] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45784.252889] [] ? wake_up_atomic_t+0x30/0x30 [45784.259490] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45784.267772] [] kthread+0xd1/0xe0 [45784.273262] [] ? insert_kthread_work+0x40/0x40 [45784.280098] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.287417] [] ? insert_kthread_work+0x40/0x40 [45784.294267] ldlm_bl_02 S ffff99ec5c3bc200 0 3549 2 0x00000080 [45784.302258] Call Trace: [45784.305018] [] schedule+0x29/0x70 [45784.310648] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45784.318352] [] ? wake_up_atomic_t+0x30/0x30 [45784.324943] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45784.333222] [] kthread+0xd1/0xe0 [45784.338718] [] ? insert_kthread_work+0x40/0x40 [45784.345556] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.352910] [] ? insert_kthread_work+0x40/0x40 [45784.359745] ldlm_elt S ffff99ec5c3bd280 0 3550 2 0x00000080 [45784.367707] Call Trace: [45784.370460] [] schedule+0x29/0x70 [45784.376093] [] expired_lock_main+0x1da/0xc50 [ptlrpc] [45784.383618] [] ? wake_up_atomic_t+0x30/0x30 [45784.390214] [] ? ldlm_add_waiting_lock+0x2f0/0x2f0 [ptlrpc] [45784.398338] [] kthread+0xd1/0xe0 [45784.403823] [] ? insert_kthread_work+0x40/0x40 [45784.410658] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.417978] [] ? insert_kthread_work+0x40/0x40 [45784.424795] mgs_params_noti S ffff99ec5c3be300 0 3551 2 0x00000080 [45784.432752] Call Trace: [45784.435506] [] schedule+0x29/0x70 [45784.441103] [] mgs_ir_notify+0x1eb/0x2a0 [mgs] [45784.447942] [] ? wake_up_atomic_t+0x30/0x30 [45784.454486] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [45784.462189] [] kthread+0xd1/0xe0 [45784.467656] [] ? insert_kthread_work+0x40/0x40 [45784.474478] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.481796] [] ? insert_kthread_work+0x40/0x40 [45784.488620] ll_mgs_0000 S ffff99ec63828000 0 3552 2 0x00000080 [45784.496579] Call Trace: [45784.499348] [] schedule+0x29/0x70 [45784.504993] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.512510] [] ? wake_up_atomic_t+0x30/0x30 [45784.519114] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.526210] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.533925] [] kthread+0xd1/0xe0 [45784.539414] [] ? insert_kthread_work+0x40/0x40 [45784.546262] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.553574] [] ? insert_kthread_work+0x40/0x40 [45784.560397] ll_mgs_0001 S ffff99ec63829080 0 3553 2 0x00000080 [45784.568337] Call Trace: [45784.571105] [] schedule+0x29/0x70 [45784.576727] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.584268] [] ? wake_up_atomic_t+0x30/0x30 [45784.590870] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.597981] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.605691] [] kthread+0xd1/0xe0 [45784.611162] [] ? insert_kthread_work+0x40/0x40 [45784.617991] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.625304] [] ? insert_kthread_work+0x40/0x40 [45784.632141] ll_mgs_0002 S ffff99ec6382a100 0 3554 2 0x00000080 [45784.640095] Call Trace: [45784.642841] [] schedule+0x29/0x70 [45784.648480] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.656011] [] ? wake_up_atomic_t+0x30/0x30 [45784.662622] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.669720] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.677425] [] kthread+0xd1/0xe0 [45784.682910] [] ? insert_kthread_work+0x40/0x40 [45784.689737] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.697049] [] ? insert_kthread_work+0x40/0x40 [45784.703882] ll_evictor S ffff99ec6382b180 0 3555 2 0x00000080 [45784.711846] Call Trace: [45784.714600] [] schedule+0x29/0x70 [45784.720224] [] ping_evictor_main+0x4f6/0x6c0 [ptlrpc] [45784.727756] [] ? wake_up_atomic_t+0x30/0x30 [45784.734372] [] ? ptlrpc_pinger_add_import+0x220/0x220 [ptlrpc] [45784.742766] [] kthread+0xd1/0xe0 [45784.748239] [] ? insert_kthread_work+0x40/0x40 [45784.755077] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.762389] [] ? insert_kthread_work+0x40/0x40 [45784.769222] ll_cfg_requeue S ffff99ec6382c200 0 3556 2 0x00000080 [45784.777170] Call Trace: [45784.779925] [] schedule+0x29/0x70 [45784.785499] [] mgc_requeue_thread+0x616/0x8b0 [mgc] [45784.792826] [] ? wake_up_atomic_t+0x30/0x30 [45784.799362] [] ? mgc_process_config+0xf30/0xf30 [mgc] [45784.806874] [] kthread+0xd1/0xe0 [45784.812363] [] ? insert_kthread_work+0x40/0x40 [45784.819195] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.826506] [] ? insert_kthread_work+0x40/0x40 [45784.833349] mdt00_000 S ffff99ec6382d280 0 3560 2 0x00000080 [45784.841303] Call Trace: [45784.844054] [] schedule+0x29/0x70 [45784.849681] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45784.857205] [] ? wake_up_atomic_t+0x30/0x30 [45784.863814] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45784.870911] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45784.878612] [] kthread+0xd1/0xe0 [45784.884084] [] ? insert_kthread_work+0x40/0x40 [45784.890915] [] ret_from_fork_nospec_begin+0x21/0x21 [45784.898250] [] ? insert_kthread_work+0x40/0x40 [45784.905108] mdt00_001 S ffff99ec6382e300 0 3561 2 0x00000080 [45784.913082] Call Trace: [45784.915837] [] schedule+0x29/0x70 [45784.921477] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45784.929193] [] ? wake_up_atomic_t+0x30/0x30 [45784.935792] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45784.943741] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45784.951798] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45784.959290] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45784.968105] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45784.975534] [] osp_md_object_lock+0x160/0x300 [osp] [45784.982884] [] lod_object_lock+0xe2/0x7d0 [lod] [45784.989824] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45784.997473] [] ? lu_ucred+0x1e/0x30 [obdclass] [45785.004368] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45785.012456] [] mdd_object_lock+0x30/0xd0 [mdd] [45785.019325] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45785.027426] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45785.035151] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45785.043059] [] mdt_object_lock+0x20/0x30 [mdt] [45785.049921] [] mdt_create+0x6de/0xe40 [mdt] [45785.056539] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45785.064372] [] mdt_reint_create+0x3a0/0x460 [mdt] [45785.071503] [] mdt_reint_rec+0x8a/0x240 [mdt] [45785.078249] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45785.085583] [] mdt_reint+0x67/0x150 [mdt] [45785.092008] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45785.099781] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45785.108286] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45785.116247] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45785.124838] [] ? wake_up_atomic_t+0x30/0x30 [45785.131446] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45785.138552] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.146276] [] kthread+0xd1/0xe0 [45785.151746] [] ? insert_kthread_work+0x40/0x40 [45785.158573] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.165883] [] ? insert_kthread_work+0x40/0x40 [45785.172714] mdt00_002 S ffff99ec4e598000 0 3562 2 0x00000080 [45785.180693] Call Trace: [45785.183446] [] schedule+0x29/0x70 [45785.189077] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45785.196787] [] ? wake_up_atomic_t+0x30/0x30 [45785.203385] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45785.211349] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45785.219415] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45785.226896] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45785.235679] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45785.243105] [] osp_md_object_lock+0x160/0x300 [osp] [45785.250438] [] lod_object_lock+0xe2/0x7d0 [lod] [45785.257388] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45785.265046] [] ? lu_ucred+0x1e/0x30 [obdclass] [45785.271887] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45785.279997] [] mdd_object_lock+0x30/0xd0 [mdd] [45785.286842] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45785.294939] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45785.302667] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45785.310563] [] mdt_object_lock+0x20/0x30 [mdt] [45785.317396] [] mdt_create+0x6de/0xe40 [mdt] [45785.324008] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45785.331829] [] mdt_reint_create+0x3a0/0x460 [mdt] [45785.338967] [] mdt_reint_rec+0x8a/0x240 [mdt] [45785.345704] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45785.353040] [] mdt_reint+0x67/0x150 [mdt] [45785.359467] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45785.367243] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45785.375736] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45785.383681] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45785.392307] [] ? wake_up_atomic_t+0x30/0x30 [45785.398915] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45785.405987] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.413697] [] kthread+0xd1/0xe0 [45785.419186] [] ? insert_kthread_work+0x40/0x40 [45785.426014] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.433334] [] ? insert_kthread_work+0x40/0x40 [45785.440192] mdt01_000 S ffff99ec4e599080 0 3563 2 0x00000080 [45785.448175] Call Trace: [45785.450941] [] schedule+0x29/0x70 [45785.456577] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45785.464094] [] ? wake_up_atomic_t+0x30/0x30 [45785.470698] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45785.477801] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.485497] [] kthread+0xd1/0xe0 [45785.490980] [] ? insert_kthread_work+0x40/0x40 [45785.497814] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.505131] [] ? insert_kthread_work+0x40/0x40 [45785.511962] mdt01_001 S ffff99ec4e59a100 0 3564 2 0x00000080 [45785.519928] Call Trace: [45785.522689] [] schedule+0x29/0x70 [45785.528334] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45785.535850] [] ? wake_up_atomic_t+0x30/0x30 [45785.542461] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45785.549553] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.557258] [] kthread+0xd1/0xe0 [45785.562733] [] ? insert_kthread_work+0x40/0x40 [45785.569546] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.576855] [] ? insert_kthread_work+0x40/0x40 [45785.583686] mdt01_002 S ffff99ec4e59b180 0 3565 2 0x00000080 [45785.591632] Call Trace: [45785.594404] [] schedule+0x29/0x70 [45785.600033] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45785.607744] [] ? wake_up_atomic_t+0x30/0x30 [45785.614348] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45785.622313] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45785.630391] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45785.637874] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45785.646657] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45785.654080] [] osp_md_object_lock+0x160/0x300 [osp] [45785.661433] [] lod_object_lock+0xe2/0x7d0 [lod] [45785.668381] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45785.676030] [] ? lu_ucred+0x1e/0x30 [obdclass] [45785.682900] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45785.691005] [] mdd_object_lock+0x30/0xd0 [mdd] [45785.697861] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45785.705940] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45785.713663] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45785.721571] [] mdt_object_lock+0x20/0x30 [mdt] [45785.728426] [] mdt_create+0x6de/0xe40 [mdt] [45785.735038] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45785.742860] [] mdt_reint_create+0x3a0/0x460 [mdt] [45785.750003] [] mdt_reint_rec+0x8a/0x240 [mdt] [45785.756755] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45785.764089] [] mdt_reint+0x67/0x150 [mdt] [45785.770517] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45785.778294] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45785.786791] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45785.794749] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45785.803351] [] ? task_rq_unlock+0x20/0x20 [45785.809684] [] ? __wake_up+0x13/0x20 [45785.815618] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45785.822732] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.830433] [] kthread+0xd1/0xe0 [45785.835901] [] ? insert_kthread_work+0x40/0x40 [45785.842728] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.850045] [] ? insert_kthread_work+0x40/0x40 [45785.856891] mdt_rdpg00_000 S ffff99ec4e59c200 0 3566 2 0x00000080 [45785.864858] Call Trace: [45785.867611] [] schedule+0x29/0x70 [45785.873246] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45785.880768] [] ? wake_up_atomic_t+0x30/0x30 [45785.887384] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45785.894491] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.902212] [] kthread+0xd1/0xe0 [45785.907691] [] ? insert_kthread_work+0x40/0x40 [45785.914505] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.921823] [] ? insert_kthread_work+0x40/0x40 [45785.928655] mdt_rdpg00_001 S ffff99ec4e59d280 0 3567 2 0x00000080 [45785.936610] Call Trace: [45785.939358] [] schedule+0x29/0x70 [45785.944999] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45785.952522] [] ? wake_up_atomic_t+0x30/0x30 [45785.959109] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45785.966212] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45785.973914] [] kthread+0xd1/0xe0 [45785.979359] [] ? insert_kthread_work+0x40/0x40 [45785.986179] [] ret_from_fork_nospec_begin+0x21/0x21 [45785.993491] [] ? insert_kthread_work+0x40/0x40 [45786.000337] mdt_rdpg01_000 S ffff99ec4e59e300 0 3568 2 0x00000080 [45786.008287] Call Trace: [45786.011059] [] schedule+0x29/0x70 [45786.016701] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.024216] [] ? wake_up_atomic_t+0x30/0x30 [45786.030820] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.037930] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.045633] [] kthread+0xd1/0xe0 [45786.051103] [] ? insert_kthread_work+0x40/0x40 [45786.057931] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.065253] [] ? insert_kthread_work+0x40/0x40 [45786.072090] mdt_rdpg01_001 S ffff99ec4de08000 0 3569 2 0x00000080 [45786.080077] Call Trace: [45786.082842] [] schedule+0x29/0x70 [45786.088482] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.095997] [] ? wake_up_atomic_t+0x30/0x30 [45786.102600] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.109689] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.117404] [] kthread+0xd1/0xe0 [45786.122872] [] ? insert_kthread_work+0x40/0x40 [45786.129688] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.137012] [] ? insert_kthread_work+0x40/0x40 [45786.143864] mdt_out00_000 S ffff99ec4de09080 0 3570 2 0x00000080 [45786.151824] Call Trace: [45786.154577] [] schedule+0x29/0x70 [45786.160209] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.167725] [] ? wake_up_atomic_t+0x30/0x30 [45786.174335] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.181432] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.189128] [] kthread+0xd1/0xe0 [45786.194591] [] ? insert_kthread_work+0x40/0x40 [45786.201430] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.208748] [] ? insert_kthread_work+0x40/0x40 [45786.215579] mdt_out00_001 S ffff99ec4de0a100 0 3571 2 0x00000080 [45786.223538] Call Trace: [45786.226292] [] schedule+0x29/0x70 [45786.231924] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.239429] [] ? wake_up_atomic_t+0x30/0x30 [45786.246037] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.253141] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.260850] [] kthread+0xd1/0xe0 [45786.266346] [] ? insert_kthread_work+0x40/0x40 [45786.273181] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.280491] [] ? insert_kthread_work+0x40/0x40 [45786.287350] mdt_out01_000 S ffff99ec4de0b180 0 3572 2 0x00000080 [45786.295293] Call Trace: [45786.298052] [] schedule+0x29/0x70 [45786.303683] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.311200] [] ? wake_up_atomic_t+0x30/0x30 [45786.317802] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.324906] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.332606] [] kthread+0xd1/0xe0 [45786.338081] [] ? insert_kthread_work+0x40/0x40 [45786.344908] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.352227] [] ? insert_kthread_work+0x40/0x40 [45786.359066] mdt_out01_001 S ffff99ec4de0c200 0 3573 2 0x00000080 [45786.367033] Call Trace: [45786.369785] [] schedule+0x29/0x70 [45786.375437] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.382958] [] ? wake_up_atomic_t+0x30/0x30 [45786.389607] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.396713] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.404418] [] kthread+0xd1/0xe0 [45786.409888] [] ? insert_kthread_work+0x40/0x40 [45786.416719] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.424057] [] ? insert_kthread_work+0x40/0x40 [45786.430896] mdt_seqs_0000 S ffff99ec4de0d280 0 3574 2 0x00000080 [45786.438830] Call Trace: [45786.441577] [] schedule+0x29/0x70 [45786.447208] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.454735] [] ? wake_up_atomic_t+0x30/0x30 [45786.461349] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.468459] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.476161] [] kthread+0xd1/0xe0 [45786.481632] [] ? insert_kthread_work+0x40/0x40 [45786.488458] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.495769] [] ? insert_kthread_work+0x40/0x40 [45786.502601] mdt_seqs_0001 S ffff99ec4de0e300 0 3575 2 0x00000080 [45786.510544] Call Trace: [45786.513293] [] schedule+0x29/0x70 [45786.518920] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.526455] [] ? wake_up_atomic_t+0x30/0x30 [45786.533057] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.540161] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.547863] [] kthread+0xd1/0xe0 [45786.553354] [] ? insert_kthread_work+0x40/0x40 [45786.560181] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.567492] [] ? insert_kthread_work+0x40/0x40 [45786.574345] mdt_seqm_0000 S ffff99ec49f70000 0 3576 2 0x00000080 [45786.582313] Call Trace: [45786.585079] [] schedule+0x29/0x70 [45786.590721] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.598236] [] ? wake_up_atomic_t+0x30/0x30 [45786.604838] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.611940] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.619629] [] kthread+0xd1/0xe0 [45786.625106] [] ? insert_kthread_work+0x40/0x40 [45786.631939] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.639250] [] ? insert_kthread_work+0x40/0x40 [45786.646103] mdt_seqm_0001 S ffff99ec49f71080 0 3577 2 0x00000080 [45786.654091] Call Trace: [45786.656853] [] schedule+0x29/0x70 [45786.662498] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.670008] [] ? wake_up_atomic_t+0x30/0x30 [45786.676604] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.683696] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.691392] [] kthread+0xd1/0xe0 [45786.696879] [] ? insert_kthread_work+0x40/0x40 [45786.703694] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.711019] [] ? insert_kthread_work+0x40/0x40 [45786.717857] mdt_fld_0000 S ffff99ec49f72100 0 3578 2 0x00000080 [45786.725825] Call Trace: [45786.728564] [] schedule+0x29/0x70 [45786.734202] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.741717] [] ? wake_up_atomic_t+0x30/0x30 [45786.748308] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.755428] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.763130] [] kthread+0xd1/0xe0 [45786.768600] [] ? insert_kthread_work+0x40/0x40 [45786.775452] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.782770] [] ? insert_kthread_work+0x40/0x40 [45786.789594] mdt_fld_0001 S ffff99ec49f73180 0 3579 2 0x00000080 [45786.797537] Call Trace: [45786.800302] [] schedule+0x29/0x70 [45786.805922] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.813442] [] ? wake_up_atomic_t+0x30/0x30 [45786.820052] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.827162] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.834871] [] kthread+0xd1/0xe0 [45786.840377] [] ? insert_kthread_work+0x40/0x40 [45786.847203] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.854515] [] ? insert_kthread_work+0x40/0x40 [45786.861366] mdt_io00_000 S ffff99ec49f74200 0 3580 2 0x00000080 [45786.869301] Call Trace: [45786.872064] [] schedule+0x29/0x70 [45786.877697] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.885205] [] ? wake_up_atomic_t+0x30/0x30 [45786.891808] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.898918] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.906618] [] kthread+0xd1/0xe0 [45786.912094] [] ? insert_kthread_work+0x40/0x40 [45786.918922] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.926241] [] ? insert_kthread_work+0x40/0x40 [45786.933071] mdt_io00_001 S ffff99ec49f75280 0 3581 2 0x00000080 [45786.941031] Call Trace: [45786.943792] [] schedule+0x29/0x70 [45786.949445] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45786.956959] [] ? wake_up_atomic_t+0x30/0x30 [45786.963578] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45786.970692] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45786.978403] [] kthread+0xd1/0xe0 [45786.983874] [] ? insert_kthread_work+0x40/0x40 [45786.990700] [] ret_from_fork_nospec_begin+0x21/0x21 [45786.998019] [] ? insert_kthread_work+0x40/0x40 [45787.004850] mdt_io00_002 S ffff99ec49f76300 0 3582 2 0x00000080 [45787.012795] Call Trace: [45787.015546] [] schedule+0x29/0x70 [45787.021173] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45787.028690] [] ? wake_up_atomic_t+0x30/0x30 [45787.035305] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45787.042407] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45787.050121] [] kthread+0xd1/0xe0 [45787.055581] [] ? insert_kthread_work+0x40/0x40 [45787.062408] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.069720] [] ? insert_kthread_work+0x40/0x40 [45787.076544] mdt_io01_000 S ffff99ec47818000 0 3583 2 0x00000080 [45787.084483] Call Trace: [45787.087247] [] schedule+0x29/0x70 [45787.092860] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45787.100385] [] ? wake_up_atomic_t+0x30/0x30 [45787.106994] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45787.114098] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45787.121801] [] kthread+0xd1/0xe0 [45787.127271] [] ? insert_kthread_work+0x40/0x40 [45787.134100] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.141402] [] ? insert_kthread_work+0x40/0x40 [45787.148232] mdt_io01_001 S ffff99ec47819080 0 3584 2 0x00000080 [45787.156226] Call Trace: [45787.158999] [] schedule+0x29/0x70 [45787.164623] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45787.172140] [] ? wake_up_atomic_t+0x30/0x30 [45787.178743] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45787.185845] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45787.193548] [] kthread+0xd1/0xe0 [45787.199017] [] ? insert_kthread_work+0x40/0x40 [45787.205842] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.213152] [] ? insert_kthread_work+0x40/0x40 [45787.219997] mdt_io01_002 S ffff99ec4781a100 0 3585 2 0x00000080 [45787.227986] Call Trace: [45787.230755] [] schedule+0x29/0x70 [45787.236400] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45787.243917] [] ? wake_up_atomic_t+0x30/0x30 [45787.250525] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45787.257627] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45787.265332] [] kthread+0xd1/0xe0 [45787.270801] [] ? insert_kthread_work+0x40/0x40 [45787.277630] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.284947] [] ? insert_kthread_work+0x40/0x40 [45787.291785] mgs_soaked_noti S ffff99ec4781b180 0 3586 2 0x00000080 [45787.299742] Call Trace: [45787.302511] [] schedule+0x29/0x70 [45787.308095] [] mgs_ir_notify+0x1eb/0x2a0 [mgs] [45787.314936] [] ? wake_up_atomic_t+0x30/0x30 [45787.321480] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [45787.329171] [] kthread+0xd1/0xe0 [45787.334641] [] ? insert_kthread_work+0x40/0x40 [45787.341466] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.348777] [] ? insert_kthread_work+0x40/0x40 [45787.355615] qmt_reba_soaked S ffff99ec4781e300 0 3593 2 0x00000080 [45787.363575] Call Trace: [45787.366331] [] schedule+0x29/0x70 [45787.371926] [] qmt_reba_thread+0x377/0xa70 [lquota] [45787.379233] [] ? qmt_glimpse_lock.isra.15+0xa90/0xa90 [lquota] [45787.387613] [] kthread+0xd1/0xe0 [45787.393130] [] ? insert_kthread_work+0x40/0x40 [45787.399977] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.407288] [] ? insert_kthread_work+0x40/0x40 [45787.414107] mdt_restriper_0 S ffff99ec4781c200 0 3594 2 0x00000080 [45787.422114] Call Trace: [45787.424884] [] schedule+0x29/0x70 [45787.430490] [] mdt_restriper_main+0x3a5/0x1000 [mdt] [45787.437901] [] ? __schedule+0x320/0x680 [45787.444063] [] ? mdt_auto_split+0x12e0/0x12e0 [mdt] [45787.451391] [] kthread+0xd1/0xe0 [45787.456872] [] ? insert_kthread_work+0x40/0x40 [45787.463701] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.471012] [] ? insert_kthread_work+0x40/0x40 [45787.477860] osp_up1-0 S ffff99ec5c2e8000 0 3597 2 0x00000080 [45787.485846] Call Trace: [45787.488610] [] schedule+0x29/0x70 [45787.494188] [] osp_send_update_thread+0x205/0x4a0 [osp] [45787.501893] [] ? wake_up_atomic_t+0x30/0x30 [45787.508426] [] ? osp_invalidate_request+0x3a0/0x3a0 [osp] [45787.516312] [] kthread+0xd1/0xe0 [45787.521784] [] ? insert_kthread_work+0x40/0x40 [45787.528611] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.535923] [] ? insert_kthread_work+0x40/0x40 [45787.542754] osp-pre-1-0 S ffff99ec5c2e9080 0 3598 2 0x00000080 [45787.550715] Call Trace: [45787.553490] [] schedule+0x29/0x70 [45787.559071] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45787.566680] [] ? wake_up_atomic_t+0x30/0x30 [45787.573223] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45787.580527] [] kthread+0xd1/0xe0 [45787.585995] [] ? insert_kthread_work+0x40/0x40 [45787.592823] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.600132] [] ? insert_kthread_work+0x40/0x40 [45787.606978] osp_up2-0 S ffff99ec5c2ea100 0 3599 2 0x00000080 [45787.614943] Call Trace: [45787.617705] [] schedule+0x29/0x70 [45787.623285] [] osp_send_update_thread+0x205/0x4a0 [osp] [45787.630998] [] ? wake_up_atomic_t+0x30/0x30 [45787.637535] [] ? osp_invalidate_request+0x3a0/0x3a0 [osp] [45787.645429] [] kthread+0xd1/0xe0 [45787.650900] [] ? insert_kthread_work+0x40/0x40 [45787.657730] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.665052] [] ? insert_kthread_work+0x40/0x40 [45787.671876] osp-pre-2-0 S ffff99ec5c2eb180 0 3600 2 0x00000080 [45787.679837] Call Trace: [45787.682589] [] schedule+0x29/0x70 [45787.688168] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45787.695778] [] ? wake_up_atomic_t+0x30/0x30 [45787.702320] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45787.709630] [] kthread+0xd1/0xe0 [45787.715100] [] ? insert_kthread_work+0x40/0x40 [45787.721935] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.729241] [] ? insert_kthread_work+0x40/0x40 [45787.736078] osp_up3-0 S ffff99ec5c2ec200 0 3601 2 0x00000080 [45787.744037] Call Trace: [45787.746791] [] schedule+0x29/0x70 [45787.752369] [] osp_send_update_thread+0x205/0x4a0 [osp] [45787.760103] [] ? wake_up_atomic_t+0x30/0x30 [45787.766647] [] ? osp_invalidate_request+0x3a0/0x3a0 [osp] [45787.774543] [] kthread+0xd1/0xe0 [45787.780012] [] ? insert_kthread_work+0x40/0x40 [45787.786848] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.794152] [] ? insert_kthread_work+0x40/0x40 [45787.800989] osp-pre-3-0 S ffff99ec5c2ed280 0 3602 2 0x00000080 [45787.808952] Call Trace: [45787.811705] [] schedule+0x29/0x70 [45787.817285] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45787.824888] [] ? wake_up_atomic_t+0x30/0x30 [45787.831439] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45787.838762] [] kthread+0xd1/0xe0 [45787.844231] [] ? insert_kthread_work+0x40/0x40 [45787.851059] [] ret_from_fork_nospec_begin+0x21/0x21 [45787.858371] [] ? insert_kthread_work+0x40/0x40 [45787.865225] osp-syn-0-0 S ffff99ec5c2ee300 0 3603 2 0x00000080 [45787.873197] Call Trace: [45787.875963] [] schedule+0x29/0x70 [45787.881539] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45787.889342] [] ? wake_up_atomic_t+0x30/0x30 [45787.895912] [] llog_process_thread+0x913/0x1c10 [obdclass] [45787.903959] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45787.911971] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45787.920182] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45787.928103] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45787.936606] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45787.944520] [] llog_process_thread+0x913/0x1c10 [obdclass] [45787.952515] [] ? probe_sched_wakeup+0x2b/0xa0 [45787.959262] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45787.967776] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45787.975688] [] ? try_to_wake_up+0x192/0x390 [45787.982251] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45787.990756] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45787.999130] [] ? default_wake_function+0x12/0x20 [45788.006152] [] ? __wake_up_common+0x82/0x120 [45788.012785] [] ? ttwu_do_wakeup+0xb5/0xe0 [45788.019119] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.027330] [] llog_cat_process+0x2e/0x30 [obdclass] [45788.034748] [] osp_sync_thread+0x19e/0xc30 [osp] [45788.041782] [] ? __schedule+0x320/0x680 [45788.047937] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45788.056223] [] kthread+0xd1/0xe0 [45788.061701] [] ? insert_kthread_work+0x40/0x40 [45788.068529] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.075842] [] ? insert_kthread_work+0x40/0x40 [45788.082665] osp-pre-0-0 S ffff99ec478b0000 0 3604 2 0x00000080 [45788.090599] Call Trace: [45788.093346] [] schedule+0x29/0x70 [45788.098918] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45788.106521] [] ? wake_up_atomic_t+0x30/0x30 [45788.113063] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45788.120385] [] kthread+0xd1/0xe0 [45788.125863] [] ? insert_kthread_work+0x40/0x40 [45788.132692] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.140004] [] ? insert_kthread_work+0x40/0x40 [45788.146836] osp-syn-1-0 S ffff99ec478b1080 0 3605 2 0x00000080 [45788.154781] Call Trace: [45788.157525] [] schedule+0x29/0x70 [45788.163099] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45788.170907] [] ? wake_up_atomic_t+0x30/0x30 [45788.177481] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.185545] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45788.193546] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.201761] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.209676] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45788.218178] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45788.226098] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.234099] [] ? probe_sched_wakeup+0x2b/0xa0 [45788.240855] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.249344] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.257240] [] ? try_to_wake_up+0x192/0x390 [45788.263802] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.272306] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45788.280697] [] ? default_wake_function+0x12/0x20 [45788.287715] [] ? __wake_up_common+0x82/0x120 [45788.294332] [] ? ttwu_do_wakeup+0xb5/0xe0 [45788.300672] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.308921] [] llog_cat_process+0x2e/0x30 [obdclass] [45788.316340] [] osp_sync_thread+0x19e/0xc30 [osp] [45788.323371] [] ? __schedule+0x320/0x680 [45788.329524] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45788.337804] [] kthread+0xd1/0xe0 [45788.343258] [] ? insert_kthread_work+0x40/0x40 [45788.350084] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.357391] [] ? insert_kthread_work+0x40/0x40 [45788.364200] osp-pre-1-0 S ffff99ec478b2100 0 3606 2 0x00000080 [45788.372129] Call Trace: [45788.374897] [] schedule+0x29/0x70 [45788.380468] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45788.388071] [] ? wake_up_atomic_t+0x30/0x30 [45788.394656] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45788.401964] [] kthread+0xd1/0xe0 [45788.407456] [] ? insert_kthread_work+0x40/0x40 [45788.414278] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.421583] [] ? insert_kthread_work+0x40/0x40 [45788.428409] osp-syn-2-0 S ffff99ec478b3180 0 3607 2 0x00000080 [45788.436394] Call Trace: [45788.439149] [] schedule+0x29/0x70 [45788.444714] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45788.452528] [] ? wake_up_atomic_t+0x30/0x30 [45788.459077] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.467108] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45788.475120] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.483334] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.491261] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45788.499770] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45788.507691] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.515683] [] ? probe_sched_wakeup+0x2b/0xa0 [45788.522454] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.530958] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.538854] [] ? try_to_wake_up+0x192/0x390 [45788.545441] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.553943] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45788.562330] [] ? default_wake_function+0x12/0x20 [45788.569336] [] ? __wake_up_common+0x82/0x120 [45788.575971] [] ? ttwu_do_wakeup+0xb5/0xe0 [45788.582319] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.590580] [] llog_cat_process+0x2e/0x30 [obdclass] [45788.597998] [] osp_sync_thread+0x19e/0xc30 [osp] [45788.605023] [] ? __schedule+0x320/0x680 [45788.611178] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45788.619467] [] kthread+0xd1/0xe0 [45788.624938] [] ? insert_kthread_work+0x40/0x40 [45788.631772] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.639076] [] ? insert_kthread_work+0x40/0x40 [45788.645908] osp-pre-2-0 S ffff99ec478b4200 0 3608 2 0x00000080 [45788.653846] Call Trace: [45788.656607] [] schedule+0x29/0x70 [45788.662189] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45788.669794] [] ? wake_up_atomic_t+0x30/0x30 [45788.676336] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45788.683647] [] kthread+0xd1/0xe0 [45788.689109] [] ? insert_kthread_work+0x40/0x40 [45788.695951] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.703269] [] ? insert_kthread_work+0x40/0x40 [45788.710099] osp-syn-3-0 S ffff99ec478b5280 0 3609 2 0x00000080 [45788.718034] Call Trace: [45788.720786] [] schedule+0x29/0x70 [45788.726369] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45788.734165] [] ? wake_up_atomic_t+0x30/0x30 [45788.740720] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.748727] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.756944] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.764867] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45788.773371] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45788.781277] [] llog_process_thread+0x913/0x1c10 [obdclass] [45788.789308] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.797810] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45788.805706] [] ? try_to_wake_up+0x192/0x390 [45788.812267] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45788.820763] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45788.829136] [] ? default_wake_function+0x12/0x20 [45788.836149] [] ? __wake_up_common+0x82/0x120 [45788.842804] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45788.851017] [] llog_cat_process+0x2e/0x30 [obdclass] [45788.858448] [] osp_sync_thread+0x19e/0xc30 [osp] [45788.865492] [] ? __schedule+0x320/0x680 [45788.871643] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45788.879923] [] kthread+0xd1/0xe0 [45788.885394] [] ? insert_kthread_work+0x40/0x40 [45788.892230] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.899541] [] ? insert_kthread_work+0x40/0x40 [45788.906372] osp-pre-3-0 S ffff99ec478b6300 0 3610 2 0x00000080 [45788.914335] Call Trace: [45788.917075] [] schedule+0x29/0x70 [45788.922647] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45788.930257] [] ? wake_up_atomic_t+0x30/0x30 [45788.936798] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45788.944101] [] kthread+0xd1/0xe0 [45788.949564] [] ? insert_kthread_work+0x40/0x40 [45788.956386] [] ret_from_fork_nospec_begin+0x21/0x21 [45788.963710] [] ? insert_kthread_work+0x40/0x40 [45788.970541] osp-syn-4-0 S ffff99ec474e8000 0 3611 2 0x00000080 [45788.978470] Call Trace: [45788.981224] [] schedule+0x29/0x70 [45788.986797] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45788.994601] [] ? wake_up_atomic_t+0x30/0x30 [45789.001164] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.009209] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45789.017201] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.025426] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.033341] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45789.041837] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45789.049757] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.057760] [] ? probe_sched_wakeup+0x2b/0xa0 [45789.064527] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.073026] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.080935] [] ? try_to_wake_up+0x192/0x390 [45789.087526] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.096021] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45789.104384] [] ? default_wake_function+0x12/0x20 [45789.111405] [] ? __wake_up_common+0x82/0x120 [45789.118043] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.126290] [] llog_cat_process+0x2e/0x30 [obdclass] [45789.133700] [] osp_sync_thread+0x19e/0xc30 [osp] [45789.140729] [] ? __schedule+0x320/0x680 [45789.146891] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45789.155177] [] kthread+0xd1/0xe0 [45789.160646] [] ? insert_kthread_work+0x40/0x40 [45789.167486] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.174798] [] ? insert_kthread_work+0x40/0x40 [45789.181623] osp-pre-4-0 S ffff99ec474e9080 0 3612 2 0x00000080 [45789.189569] Call Trace: [45789.192330] [] schedule+0x29/0x70 [45789.197902] [] osp_precreate_thread+0x3de/0x13d0 [osp] [45789.205529] [] ? wake_up_atomic_t+0x30/0x30 [45789.212079] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45789.219395] [] kthread+0xd1/0xe0 [45789.224857] [] ? insert_kthread_work+0x40/0x40 [45789.231684] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.238984] [] ? insert_kthread_work+0x40/0x40 [45789.245808] osp-syn-5-0 S ffff99ec474ea100 0 3613 2 0x00000080 [45789.253738] Call Trace: [45789.256492] [] schedule+0x29/0x70 [45789.262063] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45789.269875] [] ? wake_up_atomic_t+0x30/0x30 [45789.276437] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.284483] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45789.292492] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.300701] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.308613] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45789.317111] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45789.325032] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.333060] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.341567] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.349477] [] ? try_to_wake_up+0x192/0x390 [45789.356032] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.364518] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45789.372901] [] ? default_wake_function+0x12/0x20 [45789.379919] [] ? __wake_up_common+0x82/0x120 [45789.386575] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.394837] [] llog_cat_process+0x2e/0x30 [obdclass] [45789.402263] [] osp_sync_thread+0x19e/0xc30 [osp] [45789.409277] [] ? __schedule+0x320/0x680 [45789.415434] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45789.423719] [] kthread+0xd1/0xe0 [45789.429173] [] ? insert_kthread_work+0x40/0x40 [45789.436002] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.443314] [] ? insert_kthread_work+0x40/0x40 [45789.450146] osp-pre-5-0 S ffff99ec474eb180 0 3614 2 0x00000080 [45789.458092] Call Trace: [45789.460844] [] schedule+0x29/0x70 [45789.466422] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45789.474012] [] ? wake_up_atomic_t+0x30/0x30 [45789.480558] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45789.487874] [] kthread+0xd1/0xe0 [45789.493341] [] ? insert_kthread_work+0x40/0x40 [45789.500168] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.507501] [] ? insert_kthread_work+0x40/0x40 [45789.514323] osp-syn-6-0 S ffff99ec6acc8000 0 3615 2 0x00000080 [45789.522272] Call Trace: [45789.525041] [] schedule+0x29/0x70 [45789.530614] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45789.538427] [] ? wake_up_atomic_t+0x30/0x30 [45789.544980] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.553005] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45789.561015] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.569216] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.577138] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45789.585643] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45789.593587] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.601586] [] ? probe_sched_wakeup+0x2b/0xa0 [45789.608349] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.616843] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.624738] [] ? try_to_wake_up+0x192/0x390 [45789.631287] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.639790] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45789.648169] [] ? default_wake_function+0x12/0x20 [45789.655194] [] ? __wake_up_common+0x82/0x120 [45789.661833] [] ? ttwu_do_wakeup+0xb5/0xe0 [45789.668182] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.676426] [] llog_cat_process+0x2e/0x30 [obdclass] [45789.683838] [] osp_sync_thread+0x19e/0xc30 [osp] [45789.690868] [] ? __schedule+0x320/0x680 [45789.697022] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45789.705309] [] kthread+0xd1/0xe0 [45789.710779] [] ? insert_kthread_work+0x40/0x40 [45789.717611] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.724921] [] ? insert_kthread_work+0x40/0x40 [45789.731752] osp-pre-6-0 S ffff99ec6acca100 0 3616 2 0x00000080 [45789.739686] Call Trace: [45789.742449] [] schedule+0x29/0x70 [45789.748021] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45789.755629] [] ? wake_up_atomic_t+0x30/0x30 [45789.762170] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45789.769512] [] kthread+0xd1/0xe0 [45789.774982] [] ? insert_kthread_work+0x40/0x40 [45789.781816] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.789118] [] ? insert_kthread_work+0x40/0x40 [45789.795949] osp-syn-7-0 S ffff99ec6accc200 0 3617 2 0x00000080 [45789.803886] Call Trace: [45789.806631] [] schedule+0x29/0x70 [45789.812205] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45789.820021] [] ? wake_up_atomic_t+0x30/0x30 [45789.826594] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.834630] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45789.842659] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.850875] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.858798] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45789.867297] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45789.875218] [] llog_process_thread+0x913/0x1c10 [obdclass] [45789.883263] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.891764] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45789.899660] [] ? try_to_wake_up+0x192/0x390 [45789.906222] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45789.914725] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45789.923103] [] ? default_wake_function+0x12/0x20 [45789.930123] [] ? __wake_up_common+0x82/0x120 [45789.936753] [] ? ttwu_do_wakeup+0xb5/0xe0 [45789.943100] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45789.951348] [] llog_cat_process+0x2e/0x30 [obdclass] [45789.958765] [] osp_sync_thread+0x19e/0xc30 [osp] [45789.965785] [] ? __schedule+0x320/0x680 [45789.971947] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45789.980242] [] kthread+0xd1/0xe0 [45789.985716] [] ? insert_kthread_work+0x40/0x40 [45789.992544] [] ret_from_fork_nospec_begin+0x21/0x21 [45789.999869] [] ? insert_kthread_work+0x40/0x40 [45790.006692] osp-pre-7-0 S ffff99ec6accd280 0 3618 2 0x00000080 [45790.014628] Call Trace: [45790.017388] [] schedule+0x29/0x70 [45790.022960] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45790.030580] [] ? wake_up_atomic_t+0x30/0x30 [45790.037114] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45790.044430] [] kthread+0xd1/0xe0 [45790.049905] [] ? insert_kthread_work+0x40/0x40 [45790.056724] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.064035] [] ? insert_kthread_work+0x40/0x40 [45790.070858] osp-syn-8-0 S ffff99ec6acce300 0 3619 2 0x00000080 [45790.078794] Call Trace: [45790.081548] [] schedule+0x29/0x70 [45790.087122] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45790.094925] [] ? wake_up_atomic_t+0x30/0x30 [45790.101502] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.109544] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45790.117537] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.125742] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.133658] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45790.142166] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45790.150088] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.158087] [] ? probe_sched_wakeup+0x2b/0xa0 [45790.164842] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.173328] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.181215] [] ? try_to_wake_up+0x192/0x390 [45790.187777] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.196278] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45790.204649] [] ? default_wake_function+0x12/0x20 [45790.211667] [] ? __wake_up_common+0x82/0x120 [45790.218298] [] ? ttwu_do_wakeup+0xb5/0xe0 [45790.224647] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.232903] [] llog_cat_process+0x2e/0x30 [obdclass] [45790.240315] [] osp_sync_thread+0x19e/0xc30 [osp] [45790.247343] [] ? __schedule+0x320/0x680 [45790.253519] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45790.261796] [] kthread+0xd1/0xe0 [45790.267273] [] ? insert_kthread_work+0x40/0x40 [45790.274087] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.281403] [] ? insert_kthread_work+0x40/0x40 [45790.288236] osp-pre-8-0 S ffff99ec6acc9080 0 3620 2 0x00000080 [45790.296201] Call Trace: [45790.298953] [] schedule+0x29/0x70 [45790.304554] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45790.312162] [] ? wake_up_atomic_t+0x30/0x30 [45790.318705] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45790.326015] [] kthread+0xd1/0xe0 [45790.331521] [] ? insert_kthread_work+0x40/0x40 [45790.338346] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.345656] [] ? insert_kthread_work+0x40/0x40 [45790.352489] osp-syn-9-0 S ffff99ec6ac1e300 0 3621 2 0x00000080 [45790.360459] Call Trace: [45790.363225] [] schedule+0x29/0x70 [45790.368787] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45790.376598] [] ? wake_up_atomic_t+0x30/0x30 [45790.383160] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.391210] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45790.399255] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.407460] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.415382] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45790.423872] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45790.431793] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.439786] [] ? probe_sched_wakeup+0x2b/0xa0 [45790.446553] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.455053] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.462948] [] ? try_to_wake_up+0x192/0x390 [45790.469526] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.478034] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45790.486413] [] ? default_wake_function+0x12/0x20 [45790.493431] [] ? __wake_up_common+0x82/0x120 [45790.500063] [] ? ttwu_do_wakeup+0xb5/0xe0 [45790.506420] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.514638] [] llog_cat_process+0x2e/0x30 [obdclass] [45790.522053] [] osp_sync_thread+0x19e/0xc30 [osp] [45790.529072] [] ? __schedule+0x320/0x680 [45790.535227] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45790.543526] [] kthread+0xd1/0xe0 [45790.548993] [] ? insert_kthread_work+0x40/0x40 [45790.555828] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.563130] [] ? insert_kthread_work+0x40/0x40 [45790.569954] osp-pre-9-0 S ffff99ec6ac1d280 0 3622 2 0x00000080 [45790.577911] Call Trace: [45790.580656] [] schedule+0x29/0x70 [45790.586229] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45790.593823] [] ? sched_clock_cpu+0x85/0xc0 [45790.600262] [] ? wake_up_atomic_t+0x30/0x30 [45790.606803] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45790.614113] [] kthread+0xd1/0xe0 [45790.619600] [] ? insert_kthread_work+0x40/0x40 [45790.626433] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.633733] [] ? insert_kthread_work+0x40/0x40 [45790.640558] osp-syn-10-0 S ffff99ec6ac1c200 0 3623 2 0x00000080 [45790.648495] Call Trace: [45790.651254] [] schedule+0x29/0x70 [45790.656811] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45790.664606] [] ? wake_up_atomic_t+0x30/0x30 [45790.671163] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.679210] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45790.687229] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.695447] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.703368] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45790.711865] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45790.719785] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.727785] [] ? probe_sched_wakeup+0x2b/0xa0 [45790.734561] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.743063] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.750959] [] ? try_to_wake_up+0x192/0x390 [45790.757544] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45790.766041] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45790.774422] [] ? default_wake_function+0x12/0x20 [45790.781433] [] ? __wake_up_common+0x82/0x120 [45790.788064] [] ? ttwu_do_wakeup+0xb5/0xe0 [45790.794419] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.802648] [] llog_cat_process+0x2e/0x30 [obdclass] [45790.810058] [] osp_sync_thread+0x19e/0xc30 [osp] [45790.817097] [] ? __schedule+0x320/0x680 [45790.823251] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45790.831534] [] kthread+0xd1/0xe0 [45790.837008] [] ? insert_kthread_work+0x40/0x40 [45790.843833] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.851145] [] ? insert_kthread_work+0x40/0x40 [45790.857978] osp-pre-10-0 S ffff99ec6ac1b180 0 3624 2 0x00000080 [45790.865916] Call Trace: [45790.868678] [] schedule+0x29/0x70 [45790.874258] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45790.881872] [] ? wake_up_atomic_t+0x30/0x30 [45790.888414] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45790.895731] [] kthread+0xd1/0xe0 [45790.901193] [] ? insert_kthread_work+0x40/0x40 [45790.908021] [] ret_from_fork_nospec_begin+0x21/0x21 [45790.915330] [] ? insert_kthread_work+0x40/0x40 [45790.922159] osp-syn-11-0 S ffff99ec6ac1a100 0 3625 2 0x00000080 [45790.930097] Call Trace: [45790.932857] [] schedule+0x29/0x70 [45790.938439] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45790.946254] [] ? wake_up_atomic_t+0x30/0x30 [45790.952816] [] llog_process_thread+0x913/0x1c10 [obdclass] [45790.960854] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45790.968841] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45790.977059] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45790.984982] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45790.993487] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45791.001415] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.009440] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.017946] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.025835] [] ? try_to_wake_up+0x192/0x390 [45791.032398] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.040898] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45791.049284] [] ? default_wake_function+0x12/0x20 [45791.056303] [] ? __wake_up_common+0x82/0x120 [45791.062951] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.071178] [] llog_cat_process+0x2e/0x30 [obdclass] [45791.078613] [] osp_sync_thread+0x19e/0xc30 [osp] [45791.085649] [] ? __schedule+0x320/0x680 [45791.091804] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45791.100085] [] kthread+0xd1/0xe0 [45791.105576] [] ? insert_kthread_work+0x40/0x40 [45791.112406] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.119717] [] ? insert_kthread_work+0x40/0x40 [45791.126559] osp-pre-11-0 S ffff99ec6ac19080 0 3626 2 0x00000080 [45791.134521] Call Trace: [45791.137289] [] schedule+0x29/0x70 [45791.142861] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45791.150463] [] ? wake_up_atomic_t+0x30/0x30 [45791.156999] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45791.164315] [] kthread+0xd1/0xe0 [45791.169782] [] ? insert_kthread_work+0x40/0x40 [45791.176622] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.183925] [] ? insert_kthread_work+0x40/0x40 [45791.190761] osp-syn-12-0 S ffff99ec6ac18000 0 3627 2 0x00000080 [45791.198696] Call Trace: [45791.201466] [] schedule+0x29/0x70 [45791.207040] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45791.214850] [] ? wake_up_atomic_t+0x30/0x30 [45791.221404] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.229429] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45791.237422] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.245641] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.253558] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.261775] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45791.269694] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.277685] [] ? probe_sched_wakeup+0x2b/0xa0 [45791.284441] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.292936] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.300824] [] ? try_to_wake_up+0x192/0x390 [45791.307386] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.315886] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45791.324272] [] ? default_wake_function+0x12/0x20 [45791.331291] [] ? __wake_up_common+0x82/0x120 [45791.337921] [] ? ttwu_do_wakeup+0xb5/0xe0 [45791.344269] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.352504] [] llog_cat_process+0x2e/0x30 [obdclass] [45791.359935] [] osp_sync_thread+0x19e/0xc30 [osp] [45791.366964] [] ? __schedule+0x320/0x680 [45791.373117] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45791.381398] [] kthread+0xd1/0xe0 [45791.386875] [] ? insert_kthread_work+0x40/0x40 [45791.393707] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.401063] [] ? insert_kthread_work+0x40/0x40 [45791.407896] osp-pre-12-0 S ffff99ec6b6c6300 0 3628 2 0x00000080 [45791.415847] Call Trace: [45791.418600] [] schedule+0x29/0x70 [45791.424178] [] osp_precreate_thread+0x3de/0x13d0 [osp] [45791.431793] [] ? wake_up_atomic_t+0x30/0x30 [45791.438326] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45791.445654] [] kthread+0xd1/0xe0 [45791.451129] [] ? insert_kthread_work+0x40/0x40 [45791.457963] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.465274] [] ? insert_kthread_work+0x40/0x40 [45791.472103] osp-syn-13-0 S ffff99ec6b6c5280 0 3629 2 0x00000080 [45791.480047] Call Trace: [45791.482799] [] schedule+0x29/0x70 [45791.488371] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45791.496187] [] ? wake_up_atomic_t+0x30/0x30 [45791.502748] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.510780] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45791.518777] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.526993] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.534907] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45791.543409] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45791.551329] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.559376] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.567878] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.575773] [] ? try_to_wake_up+0x192/0x390 [45791.582350] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.590844] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45791.599218] [] ? default_wake_function+0x12/0x20 [45791.606236] [] ? __wake_up_common+0x82/0x120 [45791.612867] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.621107] [] llog_cat_process+0x2e/0x30 [obdclass] [45791.628527] [] osp_sync_thread+0x19e/0xc30 [osp] [45791.635581] [] ? __schedule+0x320/0x680 [45791.641743] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45791.650037] [] kthread+0xd1/0xe0 [45791.655505] [] ? insert_kthread_work+0x40/0x40 [45791.662334] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.669653] [] ? insert_kthread_work+0x40/0x40 [45791.676483] osp-pre-13-0 S ffff99ec6b6c4200 0 3630 2 0x00000080 [45791.684445] Call Trace: [45791.687199] [] schedule+0x29/0x70 [45791.692765] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45791.700379] [] ? wake_up_atomic_t+0x30/0x30 [45791.706912] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45791.714229] [] kthread+0xd1/0xe0 [45791.719698] [] ? insert_kthread_work+0x40/0x40 [45791.726525] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.733823] [] ? insert_kthread_work+0x40/0x40 [45791.740666] osp-syn-14-0 S ffff99ec6b6c3180 0 3631 2 0x00000080 [45791.748604] Call Trace: [45791.751366] [] schedule+0x29/0x70 [45791.756932] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45791.764741] [] ? wake_up_atomic_t+0x30/0x30 [45791.771311] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.779346] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45791.787352] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.795589] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.803509] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45791.812011] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45791.819934] [] llog_process_thread+0x913/0x1c10 [obdclass] [45791.827938] [] ? probe_sched_wakeup+0x2b/0xa0 [45791.834708] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.843203] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45791.851081] [] ? try_to_wake_up+0x192/0x390 [45791.857647] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45791.866148] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45791.874528] [] ? default_wake_function+0x12/0x20 [45791.881549] [] ? __wake_up_common+0x82/0x120 [45791.888188] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45791.896410] [] llog_cat_process+0x2e/0x30 [obdclass] [45791.903821] [] osp_sync_thread+0x19e/0xc30 [osp] [45791.910854] [] ? __schedule+0x320/0x680 [45791.917010] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45791.925282] [] kthread+0xd1/0xe0 [45791.930751] [] ? insert_kthread_work+0x40/0x40 [45791.937600] [] ret_from_fork_nospec_begin+0x21/0x21 [45791.944913] [] ? insert_kthread_work+0x40/0x40 [45791.951738] osp-pre-14-0 S ffff99e83c2c4200 0 3632 2 0x00000080 [45791.959680] Call Trace: [45791.962449] [] schedule+0x29/0x70 [45791.968029] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45791.975633] [] ? wake_up_atomic_t+0x30/0x30 [45791.982181] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45791.989483] [] kthread+0xd1/0xe0 [45791.994952] [] ? insert_kthread_work+0x40/0x40 [45792.001779] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.009089] [] ? insert_kthread_work+0x40/0x40 [45792.015906] osp-syn-15-0 S ffff99e83c2c5280 0 3633 2 0x00000080 [45792.023853] Call Trace: [45792.026610] [] schedule+0x29/0x70 [45792.032192] [] osp_sync_process_queues+0xbbf/0xde0 [osp] [45792.040000] [] ? wake_up_atomic_t+0x30/0x30 [45792.046565] [] llog_process_thread+0x913/0x1c10 [obdclass] [45792.054629] [] ? llog_cat_id2handle+0x3b8/0x670 [obdclass] [45792.062634] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45792.070861] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45792.078774] [] ? llog_cat_process_common+0x121/0x470 [obdclass] [45792.087278] [] llog_cat_process_cb+0x2b1/0x2c0 [obdclass] [45792.095212] [] llog_process_thread+0x913/0x1c10 [obdclass] [45792.103237] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45792.111745] [] llog_process_or_fork+0xd9/0x550 [obdclass] [45792.119639] [] ? try_to_wake_up+0x192/0x390 [45792.126212] [] ? llog_cat_cancel_records+0x1d0/0x1d0 [obdclass] [45792.134715] [] llog_cat_process_or_fork+0x1f1/0x390 [obdclass] [45792.143097] [] ? default_wake_function+0x12/0x20 [45792.150114] [] ? __wake_up_common+0x82/0x120 [45792.156759] [] ? osp_sync_process_record+0x1040/0x1040 [osp] [45792.165007] [] llog_cat_process+0x2e/0x30 [obdclass] [45792.172419] [] osp_sync_thread+0x19e/0xc30 [osp] [45792.179442] [] ? __schedule+0x320/0x680 [45792.185619] [] ? osp_sync_process_committed+0xd70/0xd70 [osp] [45792.193891] [] kthread+0xd1/0xe0 [45792.199359] [] ? insert_kthread_work+0x40/0x40 [45792.206188] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.213501] [] ? insert_kthread_work+0x40/0x40 [45792.220332] osp-pre-15-0 S ffff99e83c2c6300 0 3634 2 0x00000080 [45792.228312] Call Trace: [45792.231066] [] schedule+0x29/0x70 [45792.236636] [] osp_precreate_thread+0x7fa/0x13d0 [osp] [45792.244243] [] ? wake_up_atomic_t+0x30/0x30 [45792.250784] [] ? osp_init_pre_fid+0x680/0x680 [osp] [45792.258100] [] kthread+0xd1/0xe0 [45792.263563] [] ? insert_kthread_work+0x40/0x40 [45792.270391] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.277704] [] ? insert_kthread_work+0x40/0x40 [45792.284541] lquota_wb_soake S ffff99ec47671080 0 3639 2 0x00000080 [45792.292525] Call Trace: [45792.295290] [] schedule+0x29/0x70 [45792.300855] [] schedule_timeout+0x168/0x2d0 [45792.307396] [] ? requeue_timers+0x170/0x170 [45792.313943] [] qsd_upd_thread+0xda5/0x1190 [lquota] [45792.321267] [] ? __schedule+0x320/0x680 [45792.327422] [] ? qsd_bump_version+0x420/0x420 [lquota] [45792.335022] [] kthread+0xd1/0xe0 [45792.340499] [] ? insert_kthread_work+0x40/0x40 [45792.347331] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.354650] [] ? insert_kthread_work+0x40/0x40 [45792.361492] lquota_wb_soake S ffff99ec47672100 0 3640 2 0x00000080 [45792.369466] Call Trace: [45792.372218] [] schedule+0x29/0x70 [45792.377782] [] schedule_timeout+0x168/0x2d0 [45792.384320] [] ? requeue_timers+0x170/0x170 [45792.390863] [] qsd_upd_thread+0xda5/0x1190 [lquota] [45792.398193] [] ? __schedule+0x320/0x680 [45792.404405] [] ? qsd_bump_version+0x420/0x420 [lquota] [45792.412010] [] kthread+0xd1/0xe0 [45792.417487] [] ? insert_kthread_work+0x40/0x40 [45792.424313] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.431636] [] ? insert_kthread_work+0x40/0x40 [45792.438481] dist_txn-0 D ffff99ec47673180 0 3641 2 0x00000080 [45792.446458] Call Trace: [45792.449218] [] schedule+0x29/0x70 [45792.454785] [] rwsem_down_write_failed+0x215/0x3c0 [45792.462043] [] ? llog_osd_regular_fid_del_name_entry+0x288/0x5c0 [obdclass] [45792.471704] [] call_rwsem_down_write_failed+0x17/0x30 [45792.479222] [] down_write+0x2d/0x3d [45792.485011] [] llog_cancel_arr_rec+0x14e/0xa40 [obdclass] [45792.492932] [] ? osd_trans_stop+0x2df/0x6f0 [osd_ldiskfs] [45792.500853] [] llog_cat_cancel_arr_rec+0xd4/0x490 [obdclass] [45792.509058] [] llog_cat_cancel_records+0x77/0x1d0 [obdclass] [45792.517327] [] distribute_txn_commit_thread+0x4c7/0xa90 [ptlrpc] [45792.525989] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [45792.533688] [] kthread+0xd1/0xe0 [45792.539163] [] ? insert_kthread_work+0x40/0x40 [45792.545998] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.553317] [] ? insert_kthread_work+0x40/0x40 [45792.560147] mdt_out00_002 S ffff99ec47699080 0 3653 2 0x00000080 [45792.568096] Call Trace: [45792.570851] [] schedule+0x29/0x70 [45792.576489] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45792.583999] [] ? wake_up_atomic_t+0x30/0x30 [45792.590625] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45792.597725] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45792.605427] [] kthread+0xd1/0xe0 [45792.610894] [] ? insert_kthread_work+0x40/0x40 [45792.617724] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.625028] [] ? insert_kthread_work+0x40/0x40 [45792.631860] mdt00_003 S ffff99ec4769a100 0 3654 2 0x00000080 [45792.639813] Call Trace: [45792.642577] [] schedule+0x29/0x70 [45792.648199] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45792.655902] [] ? wake_up_atomic_t+0x30/0x30 [45792.662516] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45792.670453] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45792.678512] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45792.685988] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45792.694772] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45792.702198] [] osp_md_object_lock+0x160/0x300 [osp] [45792.709524] [] lod_object_lock+0xe2/0x7d0 [lod] [45792.716462] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45792.724113] [] ? lu_ucred+0x1e/0x30 [obdclass] [45792.730980] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45792.739096] [] mdd_object_lock+0x30/0xd0 [mdd] [45792.745938] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45792.754028] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45792.761750] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45792.769664] [] mdt_object_lock+0x20/0x30 [mdt] [45792.776532] [] mdt_create+0x6de/0xe40 [mdt] [45792.783143] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45792.790969] [] mdt_reint_create+0x3a0/0x460 [mdt] [45792.798106] [] mdt_reint_rec+0x8a/0x240 [mdt] [45792.804852] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45792.812189] [] mdt_reint+0x67/0x150 [mdt] [45792.818599] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45792.826386] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45792.834862] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45792.842834] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45792.851425] [] ? wake_up_atomic_t+0x30/0x30 [45792.858038] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45792.865153] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45792.872856] [] kthread+0xd1/0xe0 [45792.878325] [] ? insert_kthread_work+0x40/0x40 [45792.885147] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.892449] [] ? insert_kthread_work+0x40/0x40 [45792.899290] mdt01_003 S ffff99ec4769b180 0 3693 2 0x00000080 [45792.907256] Call Trace: [45792.909993] [] schedule+0x29/0x70 [45792.915649] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45792.923175] [] ? wake_up_atomic_t+0x30/0x30 [45792.929825] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45792.936930] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45792.944648] [] kthread+0xd1/0xe0 [45792.950132] [] ? insert_kthread_work+0x40/0x40 [45792.956953] [] ret_from_fork_nospec_begin+0x21/0x21 [45792.964265] [] ? insert_kthread_work+0x40/0x40 [45792.971098] mdt00_004 S ffff99ec4769c200 0 3767 2 0x00000080 [45792.979051] Call Trace: [45792.981803] [] schedule+0x29/0x70 [45792.987444] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45792.994952] [] ? wake_up_atomic_t+0x30/0x30 [45793.001546] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.008654] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.016355] [] kthread+0xd1/0xe0 [45793.021818] [] ? insert_kthread_work+0x40/0x40 [45793.028656] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.035967] [] ? insert_kthread_work+0x40/0x40 [45793.042790] mdt_out01_002 S ffff99ec4769d280 0 3768 2 0x00000080 [45793.050760] Call Trace: [45793.053515] [] schedule+0x29/0x70 [45793.059137] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.066667] [] ? wake_up_atomic_t+0x30/0x30 [45793.073268] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.080365] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.088068] [] kthread+0xd1/0xe0 [45793.093522] [] ? insert_kthread_work+0x40/0x40 [45793.100348] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.107661] [] ? insert_kthread_work+0x40/0x40 [45793.114500] mdt01_004 S ffff99ec4769e300 0 3769 2 0x00000080 [45793.122455] Call Trace: [45793.125216] [] schedule+0x29/0x70 [45793.130850] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.138361] [] ? wake_up_atomic_t+0x30/0x30 [45793.144966] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.152043] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.159744] [] kthread+0xd1/0xe0 [45793.165213] [] ? insert_kthread_work+0x40/0x40 [45793.172046] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.179365] [] ? insert_kthread_work+0x40/0x40 [45793.186201] mdt01_005 S ffff99ec6a533180 0 3770 2 0x00000080 [45793.194147] Call Trace: [45793.196899] [] schedule+0x29/0x70 [45793.202530] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.210043] [] ? wake_up_atomic_t+0x30/0x30 [45793.216656] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.223759] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.231462] [] kthread+0xd1/0xe0 [45793.236931] [] ? insert_kthread_work+0x40/0x40 [45793.243751] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.251062] [] ? insert_kthread_work+0x40/0x40 [45793.257886] mdt_rdpg01_002 S ffff99ec6a531080 0 3771 2 0x00000080 [45793.265834] Call Trace: [45793.268581] [] schedule+0x29/0x70 [45793.274214] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.281742] [] ? wake_up_atomic_t+0x30/0x30 [45793.288344] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.295447] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.303162] [] kthread+0xd1/0xe0 [45793.308618] [] ? insert_kthread_work+0x40/0x40 [45793.315447] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.322758] [] ? insert_kthread_work+0x40/0x40 [45793.329581] mdt_out01_003 S ffff99ec6a536300 0 3772 2 0x00000080 [45793.337548] Call Trace: [45793.340301] [] schedule+0x29/0x70 [45793.345933] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.353434] [] ? wake_up_atomic_t+0x30/0x30 [45793.360037] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.367142] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.374856] [] kthread+0xd1/0xe0 [45793.380352] [] ? insert_kthread_work+0x40/0x40 [45793.387179] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.394494] [] ? insert_kthread_work+0x40/0x40 [45793.401324] mdt_out01_004 S ffff99ec6a532100 0 3773 2 0x00000080 [45793.409293] Call Trace: [45793.412052] [] schedule+0x29/0x70 [45793.417693] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45793.425209] [] ? wake_up_atomic_t+0x30/0x30 [45793.431819] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45793.438930] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.446650] [] kthread+0xd1/0xe0 [45793.452119] [] ? insert_kthread_work+0x40/0x40 [45793.458937] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.466250] [] ? insert_kthread_work+0x40/0x40 [45793.473081] mdt01_006 D ffff99ec6a4b6300 0 3774 2 0x00000080 [45793.481046] Call Trace: [45793.483807] [] schedule+0x29/0x70 [45793.489373] [] rwsem_down_write_failed+0x215/0x3c0 [45793.496594] [] call_rwsem_down_write_failed+0x17/0x30 [45793.504187] [] ? sub_updates_write+0x200/0xe3b [ptlrpc] [45793.511880] [] down_write+0x2d/0x3d [45793.517685] [] llog_cat_add_rec+0x11d/0x980 [obdclass] [45793.525308] [] llog_add+0x182/0x1f0 [obdclass] [45793.532170] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [45793.540259] [] sub_updates_write+0x302/0xe3b [ptlrpc] [45793.547857] [] ? prepare_writing_updates.isra.10+0x139/0x14a [ptlrpc] [45793.556998] [] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [45793.564223] [] ? mutex_lock+0x12/0x2f [45793.570201] [] ? lod_attr_set+0x667/0xb40 [lod] [45793.577145] [] lod_trans_stop+0x25c/0x340 [lod] [45793.584088] [] mdd_trans_stop+0x2e/0x174 [mdd] [45793.590923] [] mdd_create+0x154a/0x1cd0 [mdd] [45793.597688] [] ? mdt_version_save+0x69/0xf0 [mdt] [45793.604831] [] mdo_create+0x46/0x48 [mdt] [45793.611200] [] mdt_create+0xab1/0xe40 [mdt] [45793.617814] [] ? ldlm_resource_putref+0x72/0x260 [ptlrpc] [45793.625755] [] mdt_reint_create+0x3a0/0x460 [mdt] [45793.632893] [] mdt_reint_rec+0x8a/0x240 [mdt] [45793.639631] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45793.646975] [] mdt_reint+0x67/0x150 [mdt] [45793.653397] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45793.661172] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45793.669672] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45793.677626] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45793.686228] [] ? task_rq_unlock+0x20/0x20 [45793.692576] [] ? __wake_up+0x13/0x20 [45793.698499] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45793.705595] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.713297] [] kthread+0xd1/0xe0 [45793.718744] [] ? insert_kthread_work+0x40/0x40 [45793.725572] [] ret_from_fork_nospec_begin+0x21/0x21 [45793.732887] [] ? insert_kthread_work+0x40/0x40 [45793.739748] mdt00_005 S ffff99ec6a4b2100 0 3775 2 0x00000080 [45793.747738] Call Trace: [45793.750490] [] schedule+0x29/0x70 [45793.756117] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45793.763829] [] ? wake_up_atomic_t+0x30/0x30 [45793.770426] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45793.778347] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45793.786406] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45793.793884] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45793.802694] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45793.810100] [] osp_md_object_lock+0x160/0x300 [osp] [45793.817433] [] lod_object_lock+0xe2/0x7d0 [lod] [45793.824371] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45793.832011] [] ? lu_ucred+0x1e/0x30 [obdclass] [45793.838871] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45793.847002] [] mdd_object_lock+0x30/0xd0 [mdd] [45793.853842] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45793.861930] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45793.869644] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45793.877568] [] mdt_object_lock+0x20/0x30 [mdt] [45793.884424] [] mdt_create+0x6de/0xe40 [mdt] [45793.891034] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45793.898856] [] mdt_reint_create+0x3a0/0x460 [mdt] [45793.905987] [] mdt_reint_rec+0x8a/0x240 [mdt] [45793.912751] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45793.920084] [] mdt_reint+0x67/0x150 [mdt] [45793.926513] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45793.934324] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45793.942816] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45793.950808] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45793.959390] [] ? wake_up_atomic_t+0x30/0x30 [45793.965990] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45793.973089] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45793.980793] [] kthread+0xd1/0xe0 [45793.986270] [] ? insert_kthread_work+0x40/0x40 [45793.993096] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.000408] [] ? insert_kthread_work+0x40/0x40 [45794.007246] mdt00_006 S ffff99ec6a4b5280 0 3776 2 0x00000080 [45794.015230] Call Trace: [45794.017990] [] schedule+0x29/0x70 [45794.023607] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.031118] [] ? wake_up_atomic_t+0x30/0x30 [45794.037714] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.044818] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.052522] [] kthread+0xd1/0xe0 [45794.057991] [] ? insert_kthread_work+0x40/0x40 [45794.064818] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.072135] [] ? insert_kthread_work+0x40/0x40 [45794.078975] ldlm_cn00_003 S ffff99ec6a4b4200 0 3777 2 0x00000080 [45794.086940] Call Trace: [45794.089704] [] schedule+0x29/0x70 [45794.095338] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.102855] [] ? wake_up_atomic_t+0x30/0x30 [45794.109452] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.116555] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.124257] [] kthread+0xd1/0xe0 [45794.129726] [] ? insert_kthread_work+0x40/0x40 [45794.136571] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.143889] [] ? insert_kthread_work+0x40/0x40 [45794.150725] mdt01_007 S ffff99ec6a4b3180 0 3778 2 0x00000080 [45794.158715] Call Trace: [45794.161477] [] schedule+0x29/0x70 [45794.167113] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.174632] [] ? wake_up_atomic_t+0x30/0x30 [45794.181218] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.188324] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.196032] [] kthread+0xd1/0xe0 [45794.201509] [] ? insert_kthread_work+0x40/0x40 [45794.208327] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.215642] [] ? insert_kthread_work+0x40/0x40 [45794.222480] mdt01_008 S ffff99ec5d1a8000 0 3779 2 0x00000080 [45794.230430] Call Trace: [45794.233191] [] schedule+0x29/0x70 [45794.238836] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.246353] [] ? wake_up_atomic_t+0x30/0x30 [45794.252957] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.260061] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.267781] [] kthread+0xd1/0xe0 [45794.273256] [] ? insert_kthread_work+0x40/0x40 [45794.280085] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.287396] [] ? insert_kthread_work+0x40/0x40 [45794.294213] mdt00_007 S ffff99ec5d1ab180 0 3780 2 0x00000080 [45794.302153] Call Trace: [45794.304890] [] schedule+0x29/0x70 [45794.310525] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.318028] [] ? wake_up_atomic_t+0x30/0x30 [45794.324638] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.331745] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.339446] [] kthread+0xd1/0xe0 [45794.344916] [] ? insert_kthread_work+0x40/0x40 [45794.351754] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.359055] [] ? insert_kthread_work+0x40/0x40 [45794.365876] mdt00_008 S ffff99ec6b976300 0 3781 2 0x00000080 [45794.373802] Call Trace: [45794.376560] [] schedule+0x29/0x70 [45794.382196] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.389722] [] ? wake_up_atomic_t+0x30/0x30 [45794.396330] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.403422] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.411125] [] kthread+0xd1/0xe0 [45794.416603] [] ? insert_kthread_work+0x40/0x40 [45794.423415] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.430725] [] ? insert_kthread_work+0x40/0x40 [45794.437554] kworker/29:1H S ffff99ec6b970000 0 3782 2 0x00000080 [45794.445559] Call Trace: [45794.448316] [] schedule+0x29/0x70 [45794.453873] [] worker_thread+0x1d9/0x3c0 [45794.460124] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45794.467433] [] kthread+0xd1/0xe0 [45794.472904] [] ? insert_kthread_work+0x40/0x40 [45794.479732] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.487044] [] ? insert_kthread_work+0x40/0x40 [45794.493868] ldlm_cn01_003 S ffff99ec6b974200 0 3783 2 0x00000080 [45794.501812] Call Trace: [45794.504573] [] schedule+0x29/0x70 [45794.510190] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.517725] [] ? wake_up_atomic_t+0x30/0x30 [45794.524332] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.531438] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.539139] [] kthread+0xd1/0xe0 [45794.544608] [] ? insert_kthread_work+0x40/0x40 [45794.551437] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.558762] [] ? insert_kthread_work+0x40/0x40 [45794.565591] mdt01_009 S ffff99ec6b972100 0 3784 2 0x00000080 [45794.573544] Call Trace: [45794.576308] [] schedule+0x29/0x70 [45794.581941] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.589464] [] ? wake_up_atomic_t+0x30/0x30 [45794.596067] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.603164] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.610866] [] kthread+0xd1/0xe0 [45794.616334] [] ? insert_kthread_work+0x40/0x40 [45794.623165] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.630485] [] ? insert_kthread_work+0x40/0x40 [45794.637323] mdt01_010 S ffff99ec6b971080 0 3785 2 0x00000080 [45794.645284] Call Trace: [45794.648026] [] schedule+0x29/0x70 [45794.653634] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45794.661344] [] ? wake_up_atomic_t+0x30/0x30 [45794.667946] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45794.675872] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45794.683920] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45794.691400] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45794.700195] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45794.707600] [] osp_md_object_lock+0x160/0x300 [osp] [45794.714941] [] lod_object_lock+0xe2/0x7d0 [lod] [45794.721880] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45794.729538] [] ? lu_ucred+0x1e/0x30 [obdclass] [45794.736379] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45794.744494] [] mdd_object_lock+0x30/0xd0 [mdd] [45794.751343] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45794.759448] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45794.767170] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45794.775082] [] mdt_object_lock+0x20/0x30 [mdt] [45794.781931] [] mdt_create+0x6de/0xe40 [mdt] [45794.788537] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45794.796365] [] mdt_reint_create+0x3a0/0x460 [mdt] [45794.803487] [] mdt_reint_rec+0x8a/0x240 [mdt] [45794.810231] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45794.817566] [] mdt_reint+0x67/0x150 [mdt] [45794.823993] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45794.831783] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45794.840278] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45794.848285] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45794.856870] [] ? wake_up_atomic_t+0x30/0x30 [45794.863474] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45794.870582] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.878271] [] kthread+0xd1/0xe0 [45794.883746] [] ? insert_kthread_work+0x40/0x40 [45794.890574] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.897886] [] ? insert_kthread_work+0x40/0x40 [45794.904740] mdt01_011 S ffff99ec69e38000 0 3787 2 0x00000080 [45794.912720] Call Trace: [45794.915473] [] schedule+0x29/0x70 [45794.921109] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45794.928623] [] ? wake_up_atomic_t+0x30/0x30 [45794.935269] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45794.942363] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45794.950059] [] kthread+0xd1/0xe0 [45794.955530] [] ? insert_kthread_work+0x40/0x40 [45794.962359] [] ret_from_fork_nospec_begin+0x21/0x21 [45794.969682] [] ? insert_kthread_work+0x40/0x40 [45794.976519] mdt_io01_003 S ffff99ec69e3e300 0 3788 2 0x00000080 [45794.984497] Call Trace: [45794.987262] [] schedule+0x29/0x70 [45794.992896] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.000413] [] ? wake_up_atomic_t+0x30/0x30 [45795.007015] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.014039] [] ? finish_task_switch+0x54/0x1c0 [45795.020940] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.028649] [] kthread+0xd1/0xe0 [45795.034124] [] ? insert_kthread_work+0x40/0x40 [45795.040957] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.048253] [] ? insert_kthread_work+0x40/0x40 [45795.055082] mdt_io01_004 S ffff99ec5c0f8000 0 3789 2 0x00000080 [45795.063019] Call Trace: [45795.065794] [] schedule+0x29/0x70 [45795.071434] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.078951] [] ? wake_up_atomic_t+0x30/0x30 [45795.085547] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.092659] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.100371] [] kthread+0xd1/0xe0 [45795.105846] [] ? insert_kthread_work+0x40/0x40 [45795.112676] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.119987] [] ? insert_kthread_work+0x40/0x40 [45795.126823] ldlm_cn00_004 S ffff99ec5c0fa100 0 3790 2 0x00000080 [45795.134802] Call Trace: [45795.137567] [] schedule+0x29/0x70 [45795.143182] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.150703] [] ? wake_up_atomic_t+0x30/0x30 [45795.157326] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.164439] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.172142] [] kthread+0xd1/0xe0 [45795.177619] [] ? insert_kthread_work+0x40/0x40 [45795.184453] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.191763] [] ? insert_kthread_work+0x40/0x40 [45795.198602] ldlm_cn01_004 S ffff99ec5c0f9080 0 3791 2 0x00000080 [45795.206558] Call Trace: [45795.209323] [] schedule+0x29/0x70 [45795.214948] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.222474] [] ? wake_up_atomic_t+0x30/0x30 [45795.229084] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.236189] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.243892] [] kthread+0xd1/0xe0 [45795.249360] [] ? insert_kthread_work+0x40/0x40 [45795.256190] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.263487] [] ? insert_kthread_work+0x40/0x40 [45795.270312] ldlm_cn00_005 S ffff99ec474ee300 0 3792 2 0x00000080 [45795.278260] Call Trace: [45795.281013] [] schedule+0x29/0x70 [45795.286654] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.294172] [] ? wake_up_atomic_t+0x30/0x30 [45795.300766] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.307877] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.315586] [] kthread+0xd1/0xe0 [45795.321056] [] ? insert_kthread_work+0x40/0x40 [45795.327884] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.335196] [] ? insert_kthread_work+0x40/0x40 [45795.342012] ldlm_cn01_005 S ffff99ec474ed280 0 3793 2 0x00000080 [45795.349966] Call Trace: [45795.352743] [] schedule+0x29/0x70 [45795.358372] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.365882] [] ? wake_up_atomic_t+0x30/0x30 [45795.372484] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.379585] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.387300] [] kthread+0xd1/0xe0 [45795.392770] [] ? insert_kthread_work+0x40/0x40 [45795.399577] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.406887] [] ? insert_kthread_work+0x40/0x40 [45795.413721] ldlm_cn00_006 S ffff99ec474ec200 0 3794 2 0x00000080 [45795.421739] Call Trace: [45795.424508] [] schedule+0x29/0x70 [45795.430142] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.437659] [] ? wake_up_atomic_t+0x30/0x30 [45795.444255] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.451360] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.459063] [] kthread+0xd1/0xe0 [45795.464533] [] ? insert_kthread_work+0x40/0x40 [45795.471358] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.478680] [] ? insert_kthread_work+0x40/0x40 [45795.485520] ldlm_cn01_006 S ffff99ec5d1ac200 0 3795 2 0x00000080 [45795.493499] Call Trace: [45795.496255] [] schedule+0x29/0x70 [45795.501893] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.509401] [] ? wake_up_atomic_t+0x30/0x30 [45795.515993] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.523098] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.530817] [] kthread+0xd1/0xe0 [45795.536286] [] ? insert_kthread_work+0x40/0x40 [45795.543106] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.550424] [] ? insert_kthread_work+0x40/0x40 [45795.557271] mdt_rdpg00_002 S ffff99ec5d1ad280 0 3796 2 0x00000080 [45795.565236] Call Trace: [45795.567985] [] schedule+0x29/0x70 [45795.573620] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.581130] [] ? wake_up_atomic_t+0x30/0x30 [45795.587721] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.594852] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.602553] [] kthread+0xd1/0xe0 [45795.608026] [] ? insert_kthread_work+0x40/0x40 [45795.614862] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.622172] [] ? insert_kthread_work+0x40/0x40 [45795.629004] ldlm_cn00_007 S ffff99ec69e3c200 0 3797 2 0x00000080 [45795.636966] Call Trace: [45795.639727] [] schedule+0x29/0x70 [45795.645376] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.652891] [] ? wake_up_atomic_t+0x30/0x30 [45795.659481] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.666597] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.674306] [] kthread+0xd1/0xe0 [45795.679787] [] ? insert_kthread_work+0x40/0x40 [45795.686614] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.693925] [] ? insert_kthread_work+0x40/0x40 [45795.700782] ldlm_cn01_007 S ffff99ec46b00000 0 3798 2 0x00000080 [45795.708755] Call Trace: [45795.711515] [] schedule+0x29/0x70 [45795.717150] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.724668] [] ? wake_up_atomic_t+0x30/0x30 [45795.731262] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.738354] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.746063] [] kthread+0xd1/0xe0 [45795.751539] [] ? insert_kthread_work+0x40/0x40 [45795.758359] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.765662] [] ? insert_kthread_work+0x40/0x40 [45795.772498] ldlm_cn00_008 S ffff99ec6511d280 0 3799 2 0x00000080 [45795.780464] Call Trace: [45795.783217] [] schedule+0x29/0x70 [45795.788850] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.796373] [] ? wake_up_atomic_t+0x30/0x30 [45795.802982] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.810094] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.817794] [] kthread+0xd1/0xe0 [45795.823251] [] ? insert_kthread_work+0x40/0x40 [45795.830072] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.837375] [] ? insert_kthread_work+0x40/0x40 [45795.844210] ldlm_cn01_008 S ffff99ec65119080 0 3800 2 0x00000080 [45795.852203] Call Trace: [45795.854949] [] schedule+0x29/0x70 [45795.860591] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.868102] [] ? wake_up_atomic_t+0x30/0x30 [45795.874702] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.881805] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.889518] [] kthread+0xd1/0xe0 [45795.894987] [] ? insert_kthread_work+0x40/0x40 [45795.901807] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.909122] [] ? insert_kthread_work+0x40/0x40 [45795.915946] mdt_rdpg00_003 S ffff99ec46bc0000 0 3801 2 0x00000080 [45795.923891] Call Trace: [45795.926651] [] schedule+0x29/0x70 [45795.932281] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45795.939805] [] ? wake_up_atomic_t+0x30/0x30 [45795.946408] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45795.953498] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45795.961199] [] kthread+0xd1/0xe0 [45795.966669] [] ? insert_kthread_work+0x40/0x40 [45795.973490] [] ret_from_fork_nospec_begin+0x21/0x21 [45795.980796] [] ? insert_kthread_work+0x40/0x40 [45795.987642] ldlm_cn01_009 S ffff99ec46bc1080 0 3802 2 0x00000080 [45795.995643] Call Trace: [45795.998399] [] schedule+0x29/0x70 [45796.004028] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.011543] [] ? wake_up_atomic_t+0x30/0x30 [45796.018131] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.025231] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.032950] [] kthread+0xd1/0xe0 [45796.038415] [] ? insert_kthread_work+0x40/0x40 [45796.045233] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.052550] [] ? insert_kthread_work+0x40/0x40 [45796.059397] ldlm_cn01_010 S ffff99ec46bc2100 0 3843 2 0x00000080 [45796.067379] Call Trace: [45796.070130] [] schedule+0x29/0x70 [45796.075768] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.083273] [] ? wake_up_atomic_t+0x30/0x30 [45796.089874] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.096989] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.104699] [] kthread+0xd1/0xe0 [45796.110162] [] ? insert_kthread_work+0x40/0x40 [45796.116987] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.124305] [] ? insert_kthread_work+0x40/0x40 [45796.131152] mdt_rdpg01_003 S ffff99ec69e39080 0 3844 2 0x00000080 [45796.139106] Call Trace: [45796.141871] [] schedule+0x29/0x70 [45796.147504] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.155020] [] ? wake_up_atomic_t+0x30/0x30 [45796.161622] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.168725] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.176423] [] kthread+0xd1/0xe0 [45796.181893] [] ? insert_kthread_work+0x40/0x40 [45796.188729] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.196039] [] ? insert_kthread_work+0x40/0x40 [45796.202872] mdt_rdpg00_004 S ffff99ec69e3b180 0 3845 2 0x00000080 [45796.210847] Call Trace: [45796.213603] [] schedule+0x29/0x70 [45796.219235] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.226753] [] ? wake_up_atomic_t+0x30/0x30 [45796.233357] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.240460] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.248157] [] kthread+0xd1/0xe0 [45796.253627] [] ? insert_kthread_work+0x40/0x40 [45796.260458] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.267786] [] ? insert_kthread_work+0x40/0x40 [45796.274616] mdt_rdpg00_005 S ffff99ec5c0fb180 0 3846 2 0x00000080 [45796.282588] Call Trace: [45796.285343] [] schedule+0x29/0x70 [45796.290962] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.298484] [] ? wake_up_atomic_t+0x30/0x30 [45796.305086] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.312196] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.319907] [] kthread+0xd1/0xe0 [45796.325367] [] ? insert_kthread_work+0x40/0x40 [45796.332193] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.339504] [] ? insert_kthread_work+0x40/0x40 [45796.346334] mdt_rdpg01_004 S ffff99ec5c0fe300 0 3847 2 0x00000080 [45796.354286] Call Trace: [45796.357040] [] schedule+0x29/0x70 [45796.362672] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.370189] [] ? wake_up_atomic_t+0x30/0x30 [45796.376808] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.383890] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.391615] [] kthread+0xd1/0xe0 [45796.397085] [] ? insert_kthread_work+0x40/0x40 [45796.403913] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.411224] [] ? insert_kthread_work+0x40/0x40 [45796.418041] mdt_rdpg01_005 S ffff99ec46b02100 0 3848 2 0x00000080 [45796.425990] Call Trace: [45796.428745] [] schedule+0x29/0x70 [45796.434378] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.441901] [] ? wake_up_atomic_t+0x30/0x30 [45796.448510] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.455612] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.463306] [] kthread+0xd1/0xe0 [45796.468804] [] ? insert_kthread_work+0x40/0x40 [45796.475631] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.482942] [] ? insert_kthread_work+0x40/0x40 [45796.489797] mdt_rdpg01_006 D ffff99ec46b01080 0 3849 2 0x00000080 [45796.497757] Call Trace: [45796.500523] [] schedule+0x29/0x70 [45796.506089] [] rwsem_down_write_failed+0x215/0x3c0 [45796.513310] [] call_rwsem_down_write_failed+0x17/0x30 [45796.520935] [] ? sub_updates_write+0x200/0xe3b [ptlrpc] [45796.528638] [] down_write+0x2d/0x3d [45796.534421] [] llog_cat_add_rec+0x11d/0x980 [obdclass] [45796.542042] [] llog_add+0x182/0x1f0 [obdclass] [45796.548916] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [45796.556993] [] sub_updates_write+0x302/0xe3b [ptlrpc] [45796.564591] [] ? prepare_writing_updates.isra.10+0x139/0x14a [ptlrpc] [45796.573740] [] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [45796.580984] [] ? lod_sub_attr_set+0x1c8/0x4f0 [lod] [45796.588290] [] lod_trans_stop+0x25c/0x340 [lod] [45796.595228] [] mdd_trans_stop+0x2e/0x174 [mdd] [45796.602071] [] mdd_attr_set+0x81c/0x1070 [mdd] [45796.608928] [] mdt_mfd_close+0x771/0xbb0 [mdt] [45796.615845] [] ? lustre_pack_reply_flags+0x72/0x1f0 [ptlrpc] [45796.624056] [] mdt_close_internal+0x141/0x240 [mdt] [45796.631388] [] mdt_close+0x291/0x900 [mdt] [45796.637901] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45796.645702] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45796.654187] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45796.662151] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45796.670734] [] ? task_rq_unlock+0x20/0x20 [45796.677074] [] ? __wake_up+0x13/0x20 [45796.683001] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45796.690106] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.697840] [] kthread+0xd1/0xe0 [45796.703308] [] ? insert_kthread_work+0x40/0x40 [45796.710136] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.717446] [] ? insert_kthread_work+0x40/0x40 [45796.724271] mdt_rdpg01_007 S ffff99ec46bc3180 0 3850 2 0x00000080 [45796.732220] Call Trace: [45796.734971] [] schedule+0x29/0x70 [45796.740612] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45796.748113] [] ? wake_up_atomic_t+0x30/0x30 [45796.754719] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45796.761842] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45796.769551] [] kthread+0xd1/0xe0 [45796.775022] [] ? insert_kthread_work+0x40/0x40 [45796.781866] [] ret_from_fork_nospec_begin+0x21/0x21 [45796.789171] [] ? insert_kthread_work+0x40/0x40 [45796.795994] mdt_rdpg01_008 D ffff99ec6ac26300 0 3851 2 0x00000080 [45796.803928] Call Trace: [45796.806688] [] schedule+0x29/0x70 [45796.812254] [] rwsem_down_write_failed+0x215/0x3c0 [45796.819482] [] call_rwsem_down_write_failed+0x17/0x30 [45796.827079] [] ? sub_updates_write+0x200/0xe3b [ptlrpc] [45796.834811] [] down_write+0x2d/0x3d [45796.840598] [] llog_cat_add_rec+0x11d/0x980 [obdclass] [45796.848226] [] llog_add+0x182/0x1f0 [obdclass] [45796.855102] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [45796.863179] [] sub_updates_write+0x302/0xe3b [ptlrpc] [45796.870761] [] ? prepare_writing_updates.isra.10+0x139/0x14a [ptlrpc] [45796.879916] [] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [45796.887163] [] ? lod_sub_attr_set+0x1c8/0x4f0 [lod] [45796.894485] [] lod_trans_stop+0x25c/0x340 [lod] [45796.901427] [] mdd_trans_stop+0x2e/0x174 [mdd] [45796.908256] [] mdd_attr_set+0x81c/0x1070 [mdd] [45796.915111] [] mdt_mfd_close+0x771/0xbb0 [mdt] [45796.922015] [] ? lustre_pack_reply_flags+0x72/0x1f0 [ptlrpc] [45796.930228] [] mdt_close_internal+0x141/0x240 [mdt] [45796.937551] [] mdt_close+0x291/0x900 [mdt] [45796.944075] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45796.951874] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45796.960365] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45796.968317] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45796.976913] [] ? task_rq_unlock+0x20/0x20 [45796.983240] [] ? __wake_up+0x13/0x20 [45796.989158] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45796.996265] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.003966] [] kthread+0xd1/0xe0 [45797.009435] [] ? insert_kthread_work+0x40/0x40 [45797.016270] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.023592] [] ? insert_kthread_work+0x40/0x40 [45797.030445] mdt_rdpg01_009 S ffff99ec6ac21080 0 3852 2 0x00000080 [45797.038421] Call Trace: [45797.041182] [] schedule+0x29/0x70 [45797.046834] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.054340] [] ? wake_up_atomic_t+0x30/0x30 [45797.060961] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.068085] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.075820] [] kthread+0xd1/0xe0 [45797.081303] [] ? insert_kthread_work+0x40/0x40 [45797.088114] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.095424] [] ? insert_kthread_work+0x40/0x40 [45797.102256] mdt_rdpg00_006 S ffff99ec6ac20000 0 3853 2 0x00000080 [45797.110215] Call Trace: [45797.112963] [] schedule+0x29/0x70 [45797.118596] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.126107] [] ? wake_up_atomic_t+0x30/0x30 [45797.132710] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.139835] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.147540] [] kthread+0xd1/0xe0 [45797.153010] [] ? insert_kthread_work+0x40/0x40 [45797.159845] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.167157] [] ? insert_kthread_work+0x40/0x40 [45797.173990] mdt_out00_003 S ffff99ec6ac25280 0 3855 2 0x00000080 [45797.181921] Call Trace: [45797.184685] [] schedule+0x29/0x70 [45797.190299] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.197826] [] ? wake_up_atomic_t+0x30/0x30 [45797.204436] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.211546] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.219248] [] kthread+0xd1/0xe0 [45797.224710] [] ? insert_kthread_work+0x40/0x40 [45797.231540] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.238851] [] ? insert_kthread_work+0x40/0x40 [45797.245682] mdt_out00_004 S ffff99ec6ac24200 0 3857 2 0x00000080 [45797.253654] Call Trace: [45797.256415] [] schedule+0x29/0x70 [45797.262040] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.269555] [] ? wake_up_atomic_t+0x30/0x30 [45797.276165] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.283256] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.290959] [] kthread+0xd1/0xe0 [45797.296428] [] ? system_call_after_swapgs+0x96/0x13a [45797.303854] [] ? insert_kthread_work+0x40/0x40 [45797.310673] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.317977] [] ? insert_kthread_work+0x40/0x40 [45797.324830] mdt_io00_003 S ffff99ec697eb180 0 4528 2 0x00000080 [45797.332797] Call Trace: [45797.335583] [] schedule+0x29/0x70 [45797.341225] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.348742] [] ? wake_up_atomic_t+0x30/0x30 [45797.355342] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.362454] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.370156] [] kthread+0xd1/0xe0 [45797.375617] [] ? insert_kthread_work+0x40/0x40 [45797.382436] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.389741] [] ? insert_kthread_work+0x40/0x40 [45797.396567] mdt_rdpg00_007 S ffff99ec697ee300 0 4642 2 0x00000080 [45797.404554] Call Trace: [45797.407312] [] schedule+0x29/0x70 [45797.412959] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.420475] [] ? wake_up_atomic_t+0x30/0x30 [45797.427077] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.434184] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.441904] [] kthread+0xd1/0xe0 [45797.447365] [] ? insert_kthread_work+0x40/0x40 [45797.454187] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.461489] [] ? insert_kthread_work+0x40/0x40 [45797.468334] mdt00_009 S ffff99ec6941d280 0 4645 2 0x00000080 [45797.476314] Call Trace: [45797.479066] [] schedule+0x29/0x70 [45797.484702] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.492219] [] ? wake_up_atomic_t+0x30/0x30 [45797.498810] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.505942] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.513644] [] kthread+0xd1/0xe0 [45797.519113] [] ? insert_kthread_work+0x40/0x40 [45797.525942] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.533254] [] ? insert_kthread_work+0x40/0x40 [45797.540086] mdt01_012 S ffff99ec6941a100 0 4646 2 0x00000080 [45797.548023] Call Trace: [45797.550787] [] schedule+0x29/0x70 [45797.556406] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.563937] [] ? wake_up_atomic_t+0x30/0x30 [45797.570540] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.577650] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.585342] [] kthread+0xd1/0xe0 [45797.590840] [] ? insert_kthread_work+0x40/0x40 [45797.597669] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.604979] [] ? insert_kthread_work+0x40/0x40 [45797.611827] mdt_rdpg01_010 S ffff99e8f5596300 0 4800 2 0x00000080 [45797.619794] Call Trace: [45797.622555] [] schedule+0x29/0x70 [45797.628175] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.635693] [] ? wake_up_atomic_t+0x30/0x30 [45797.642288] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.649399] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.657109] [] kthread+0xd1/0xe0 [45797.662587] [] ? insert_kthread_work+0x40/0x40 [45797.669414] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.676725] [] ? insert_kthread_work+0x40/0x40 [45797.683556] mdt_rdpg01_011 S ffff99e83ef18000 0 4801 2 0x00000080 [45797.691507] Call Trace: [45797.694257] [] schedule+0x29/0x70 [45797.699891] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.707468] [] ? ptlrpc_at_remove_timed+0x4e/0xb0 [ptlrpc] [45797.715462] [] ? wake_up_atomic_t+0x30/0x30 [45797.722061] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.729167] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.736874] [] kthread+0xd1/0xe0 [45797.742341] [] ? insert_kthread_work+0x40/0x40 [45797.749168] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.756486] [] ? insert_kthread_work+0x40/0x40 [45797.763310] mdt_out01_005 S ffff99ec46b04200 0 4805 2 0x00000080 [45797.771273] Call Trace: [45797.774025] [] schedule+0x29/0x70 [45797.779665] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.787189] [] ? wake_up_atomic_t+0x30/0x30 [45797.793794] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.800921] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.808623] [] kthread+0xd1/0xe0 [45797.814085] [] ? insert_kthread_work+0x40/0x40 [45797.820927] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.828241] [] ? insert_kthread_work+0x40/0x40 [45797.835080] mdt_out01_006 S ffff99ec46b03180 0 4806 2 0x00000080 [45797.843019] Call Trace: [45797.845784] [] schedule+0x29/0x70 [45797.851426] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.858960] [] ? wake_up_atomic_t+0x30/0x30 [45797.865570] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.872675] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.880378] [] kthread+0xd1/0xe0 [45797.885857] [] ? insert_kthread_work+0x40/0x40 [45797.892676] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.899988] [] ? insert_kthread_work+0x40/0x40 [45797.906821] mdt_out01_007 S ffff99ec6342e300 0 4874 2 0x00000080 [45797.914800] Call Trace: [45797.917573] [] schedule+0x29/0x70 [45797.923193] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45797.930711] [] ? wake_up_atomic_t+0x30/0x30 [45797.937312] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45797.944415] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45797.952118] [] kthread+0xd1/0xe0 [45797.957581] [] ? insert_kthread_work+0x40/0x40 [45797.964411] [] ret_from_fork_nospec_begin+0x21/0x21 [45797.971738] [] ? insert_kthread_work+0x40/0x40 [45797.978582] ldlm_cn01_011 S ffff99ec6342b180 0 4919 2 0x00000080 [45797.986554] Call Trace: [45797.989314] [] schedule+0x29/0x70 [45797.994952] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.002461] [] ? wake_up_atomic_t+0x30/0x30 [45798.009064] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.016167] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.023885] [] kthread+0xd1/0xe0 [45798.029363] [] ? insert_kthread_work+0x40/0x40 [45798.036200] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.043527] [] ? insert_kthread_work+0x40/0x40 [45798.050366] mdt_out01_008 S ffff99ec5c0fd280 0 4920 2 0x00000080 [45798.058335] Call Trace: [45798.061085] [] schedule+0x29/0x70 [45798.066775] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.074294] [] ? wake_up_atomic_t+0x30/0x30 [45798.080904] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.088016] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.095719] [] kthread+0xd1/0xe0 [45798.101194] [] ? insert_kthread_work+0x40/0x40 [45798.108016] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.115343] [] ? insert_kthread_work+0x40/0x40 [45798.122174] ldlm_bl_03 S ffff99ec6941b180 0 4960 2 0x00000080 [45798.130104] Call Trace: [45798.132881] [] schedule+0x29/0x70 [45798.138511] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45798.146220] [] ? wake_up_atomic_t+0x30/0x30 [45798.152812] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45798.161098] [] kthread+0xd1/0xe0 [45798.166569] [] ? insert_kthread_work+0x40/0x40 [45798.173397] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.180708] [] ? insert_kthread_work+0x40/0x40 [45798.187546] ldlm_cb01_002 S ffff99ec47676300 0 4961 2 0x00000080 [45798.195497] Call Trace: [45798.198249] [] schedule+0x29/0x70 [45798.203897] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.211398] [] ? wake_up_atomic_t+0x30/0x30 [45798.218008] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.225112] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.232800] [] kthread+0xd1/0xe0 [45798.238269] [] ? insert_kthread_work+0x40/0x40 [45798.245096] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.252407] [] ? insert_kthread_work+0x40/0x40 [45798.259232] kworker/22:1H S ffff99e86e718000 0 5119 2 0x00000080 [45798.267179] Call Trace: [45798.269932] [] schedule+0x29/0x70 [45798.275491] [] worker_thread+0x1d9/0x3c0 [45798.281734] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45798.289043] [] kthread+0xd1/0xe0 [45798.294518] [] ? insert_kthread_work+0x40/0x40 [45798.301337] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.308647] [] ? insert_kthread_work+0x40/0x40 [45798.315469] ldlm_bl_04 S ffff99e86e71a100 0 5123 2 0x00000080 [45798.323429] Call Trace: [45798.326171] [] schedule+0x29/0x70 [45798.331804] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45798.339508] [] ? wake_up_atomic_t+0x30/0x30 [45798.346109] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45798.354393] [] kthread+0xd1/0xe0 [45798.359860] [] ? insert_kthread_work+0x40/0x40 [45798.366695] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.374006] [] ? insert_kthread_work+0x40/0x40 [45798.380841] ldlm_bl_05 S ffff99ec46b06300 0 5124 2 0x00000080 [45798.388791] Call Trace: [45798.391544] [] schedule+0x29/0x70 [45798.397173] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45798.404898] [] ? wake_up_atomic_t+0x30/0x30 [45798.411496] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45798.419788] [] kthread+0xd1/0xe0 [45798.425264] [] ? insert_kthread_work+0x40/0x40 [45798.432100] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.439417] [] ? insert_kthread_work+0x40/0x40 [45798.446240] kworker/19:1H S ffff99e86d33b180 0 5126 2 0x00000080 [45798.454176] Call Trace: [45798.456936] [] schedule+0x29/0x70 [45798.462494] [] worker_thread+0x1d9/0x3c0 [45798.468738] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45798.476052] [] kthread+0xd1/0xe0 [45798.481518] [] ? insert_kthread_work+0x40/0x40 [45798.488351] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.495669] [] ? insert_kthread_work+0x40/0x40 [45798.502491] mdt01_013 S ffff99ec46bc6300 0 5286 2 0x00000080 [45798.510436] Call Trace: [45798.513188] [] schedule+0x29/0x70 [45798.518816] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.526322] [] ? wake_up_atomic_t+0x30/0x30 [45798.532916] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.540027] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.547735] [] kthread+0xd1/0xe0 [45798.553213] [] ? insert_kthread_work+0x40/0x40 [45798.560047] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.567356] [] ? insert_kthread_work+0x40/0x40 [45798.574179] mdt00_010 S ffff99ec46bc4200 0 5287 2 0x00000080 [45798.582130] Call Trace: [45798.584897] [] schedule+0x29/0x70 [45798.590521] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45798.598282] [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] [45798.606666] [] ? wake_up_atomic_t+0x30/0x30 [45798.613273] [] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [45798.621327] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45798.630112] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45798.637442] [] mdt_object_local_lock+0x52f/0xba0 [mdt] [45798.645064] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45798.652437] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45798.661207] [] mdt_object_lock_internal+0x70/0x390 [mdt] [45798.669019] [] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [45798.676796] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [45798.684074] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [45798.691899] [] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [45798.699226] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [45798.706167] [] ? mdt_getattr_name+0x2c0/0x2c0 [mdt] [45798.713501] [] mdt_intent_policy+0x1a1/0x360 [mdt] [45798.720780] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [45798.728299] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [45798.736302] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [45798.743499] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [45798.751479] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [45798.759263] [] tgt_enqueue+0x64/0x240 [ptlrpc] [45798.766170] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45798.773954] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45798.782430] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45798.790391] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45798.798984] [] ? task_rq_unlock+0x20/0x20 [45798.805330] [] ? __wake_up+0x13/0x20 [45798.811263] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45798.818339] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.826041] [] kthread+0xd1/0xe0 [45798.831511] [] ? insert_kthread_work+0x40/0x40 [45798.838331] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.845636] [] ? insert_kthread_work+0x40/0x40 [45798.852469] mdt00_011 S ffff99ec46bc5280 0 5288 2 0x00000080 [45798.860438] Call Trace: [45798.863192] [] schedule+0x29/0x70 [45798.868843] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.876370] [] ? wake_up_atomic_t+0x30/0x30 [45798.882985] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.890088] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.897775] [] kthread+0xd1/0xe0 [45798.903244] [] ? insert_kthread_work+0x40/0x40 [45798.910062] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.917377] [] ? insert_kthread_work+0x40/0x40 [45798.924211] mdt01_014 S ffff99e8f554b180 0 5289 2 0x00000080 [45798.932155] Call Trace: [45798.934928] [] schedule+0x29/0x70 [45798.940569] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45798.948079] [] ? wake_up_atomic_t+0x30/0x30 [45798.954681] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45798.961784] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45798.969486] [] kthread+0xd1/0xe0 [45798.974946] [] ? insert_kthread_work+0x40/0x40 [45798.981777] [] ret_from_fork_nospec_begin+0x21/0x21 [45798.989089] [] ? insert_kthread_work+0x40/0x40 [45798.995936] mdt00_012 S ffff99ec62ca6300 0 5290 2 0x00000080 [45799.003929] Call Trace: [45799.006698] [] schedule+0x29/0x70 [45799.012334] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.019853] [] ? wake_up_atomic_t+0x30/0x30 [45799.026456] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.033559] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.041262] [] kthread+0xd1/0xe0 [45799.046730] [] ? insert_kthread_work+0x40/0x40 [45799.053548] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.060871] [] ? insert_kthread_work+0x40/0x40 [45799.067683] ldlm_cn01_012 S ffff99ec6342a100 0 5445 2 0x00000080 [45799.075728] Call Trace: [45799.078494] [] schedule+0x29/0x70 [45799.084128] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.091637] [] ? wake_up_atomic_t+0x30/0x30 [45799.098230] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.105336] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.113023] [] kthread+0xd1/0xe0 [45799.118486] [] ? insert_kthread_work+0x40/0x40 [45799.125324] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.132634] [] ? insert_kthread_work+0x40/0x40 [45799.139464] ldlm_cn01_013 S ffff99ec63428000 0 5599 2 0x00000080 [45799.147434] Call Trace: [45799.150186] [] schedule+0x29/0x70 [45799.155819] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.163337] [] ? wake_up_atomic_t+0x30/0x30 [45799.169942] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.177059] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.184767] [] kthread+0xd1/0xe0 [45799.190236] [] ? insert_kthread_work+0x40/0x40 [45799.197070] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.204382] [] ? insert_kthread_work+0x40/0x40 [45799.211215] mdt_rdpg00_008 S ffff99ec69e3a100 0 5764 2 0x00000080 [45799.219167] Call Trace: [45799.221932] [] schedule+0x29/0x70 [45799.227567] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.235084] [] ? wake_up_atomic_t+0x30/0x30 [45799.241677] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.248779] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.256497] [] kthread+0xd1/0xe0 [45799.261986] [] ? insert_kthread_work+0x40/0x40 [45799.268804] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.276113] [] ? insert_kthread_work+0x40/0x40 [45799.282937] ldlm_bl_06 S ffff99ec6cb16300 0 5766 2 0x00000080 [45799.290880] Call Trace: [45799.293635] [] schedule+0x29/0x70 [45799.299260] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45799.306982] [] ? wake_up_atomic_t+0x30/0x30 [45799.313581] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45799.321874] [] kthread+0xd1/0xe0 [45799.327357] [] ? insert_kthread_work+0x40/0x40 [45799.334186] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.341497] [] ? insert_kthread_work+0x40/0x40 [45799.348336] kworker/24:1H S ffff99ec6cb13180 0 6411 2 0x00000080 [45799.356304] Call Trace: [45799.359058] [] schedule+0x29/0x70 [45799.364624] [] worker_thread+0x1d9/0x3c0 [45799.370870] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45799.378180] [] kthread+0xd1/0xe0 [45799.383656] [] ? insert_kthread_work+0x40/0x40 [45799.390484] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.397795] [] ? insert_kthread_work+0x40/0x40 [45799.404626] mdt_out01_009 S ffff99ec6d824200 0 6643 2 0x00000080 [45799.412591] Call Trace: [45799.415350] [] schedule+0x29/0x70 [45799.420997] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.428512] [] ? wake_up_atomic_t+0x30/0x30 [45799.435115] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.442218] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.449946] [] kthread+0xd1/0xe0 [45799.455413] [] ? insert_kthread_work+0x40/0x40 [45799.462243] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.469559] [] ? insert_kthread_work+0x40/0x40 [45799.476397] mdt01_015 S ffff99ec6d29e300 0 7035 2 0x00000080 [45799.484363] Call Trace: [45799.487111] [] schedule+0x29/0x70 [45799.492732] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.500247] [] ? wake_up_atomic_t+0x30/0x30 [45799.506858] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.513992] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.521711] [] kthread+0xd1/0xe0 [45799.527168] [] ? insert_kthread_work+0x40/0x40 [45799.534005] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.541318] [] ? insert_kthread_work+0x40/0x40 [45799.548150] mdt00_013 S ffff99e5b5332100 0 7036 2 0x00000080 [45799.556069] Call Trace: [45799.558814] [] schedule+0x29/0x70 [45799.564434] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.571960] [] ? wake_up_atomic_t+0x30/0x30 [45799.578576] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.585667] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.593368] [] kthread+0xd1/0xe0 [45799.598836] [] ? insert_kthread_work+0x40/0x40 [45799.605654] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.612958] [] ? insert_kthread_work+0x40/0x40 [45799.619781] mdt01_016 S ffff99e84dd8c200 0 7037 2 0x00000080 [45799.627731] Call Trace: [45799.630478] [] schedule+0x29/0x70 [45799.636119] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.643634] [] ? wake_up_atomic_t+0x30/0x30 [45799.650246] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.657338] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.665038] [] kthread+0xd1/0xe0 [45799.670500] [] ? insert_kthread_work+0x40/0x40 [45799.677326] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.684642] [] ? insert_kthread_work+0x40/0x40 [45799.691467] ldlm_cn01_014 S ffff99e869df8000 0 7587 2 0x00000080 [45799.699411] Call Trace: [45799.702153] [] schedule+0x29/0x70 [45799.707800] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.715318] [] ? wake_up_atomic_t+0x30/0x30 [45799.721936] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.729027] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.736730] [] kthread+0xd1/0xe0 [45799.742193] [] ? insert_kthread_work+0x40/0x40 [45799.749030] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.756361] [] ? insert_kthread_work+0x40/0x40 [45799.763187] ldlm_cn01_015 S ffff99e869dfb180 0 7588 2 0x00000080 [45799.771144] Call Trace: [45799.773892] [] schedule+0x29/0x70 [45799.779534] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.787044] [] ? wake_up_atomic_t+0x30/0x30 [45799.793638] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.800736] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.808438] [] kthread+0xd1/0xe0 [45799.813899] [] ? insert_kthread_work+0x40/0x40 [45799.820740] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.828057] [] ? insert_kthread_work+0x40/0x40 [45799.834891] ldlm_cn01_016 S ffff99e867ec8000 0 7589 2 0x00000080 [45799.842850] Call Trace: [45799.845604] [] schedule+0x29/0x70 [45799.851237] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.858745] [] ? wake_up_atomic_t+0x30/0x30 [45799.865346] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.872436] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.880138] [] kthread+0xd1/0xe0 [45799.885590] [] ? insert_kthread_work+0x40/0x40 [45799.892425] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.899748] [] ? insert_kthread_work+0x40/0x40 [45799.906578] ldlm_cn01_017 S ffff99e86f0bb180 0 7590 2 0x00000080 [45799.914533] Call Trace: [45799.917277] [] schedule+0x29/0x70 [45799.922892] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45799.930408] [] ? wake_up_atomic_t+0x30/0x30 [45799.937043] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45799.944150] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45799.951854] [] kthread+0xd1/0xe0 [45799.957328] [] ? insert_kthread_work+0x40/0x40 [45799.964162] [] ret_from_fork_nospec_begin+0x21/0x21 [45799.971490] [] ? insert_kthread_work+0x40/0x40 [45799.978314] mdt_out01_010 S ffff99e86b14b180 0 7629 2 0x00000080 [45799.986262] Call Trace: [45799.989027] [] schedule+0x29/0x70 [45799.994662] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.002170] [] ? wake_up_atomic_t+0x30/0x30 [45800.008781] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.015872] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.023581] [] kthread+0xd1/0xe0 [45800.029051] [] ? insert_kthread_work+0x40/0x40 [45800.035877] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.043190] [] ? insert_kthread_work+0x40/0x40 [45800.050028] mdt_out01_011 S ffff99e86b14a100 0 7630 2 0x00000080 [45800.057986] Call Trace: [45800.060739] [] schedule+0x29/0x70 [45800.066371] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.073879] [] ? wake_up_atomic_t+0x30/0x30 [45800.080473] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.087571] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.095280] [] kthread+0xd1/0xe0 [45800.100748] [] ? insert_kthread_work+0x40/0x40 [45800.107574] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.114887] [] ? insert_kthread_work+0x40/0x40 [45800.121713] ldlm_cn01_018 S ffff99e86b14c200 0 7669 2 0x00000080 [45800.129642] Call Trace: [45800.132396] [] schedule+0x29/0x70 [45800.138045] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.145554] [] ? wake_up_atomic_t+0x30/0x30 [45800.152165] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.159264] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.166982] [] kthread+0xd1/0xe0 [45800.172452] [] ? insert_kthread_work+0x40/0x40 [45800.179281] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.186599] [] ? insert_kthread_work+0x40/0x40 [45800.193423] mdt_out01_012 S ffff99e86b149080 0 7670 2 0x00000080 [45800.201369] Call Trace: [45800.204115] [] schedule+0x29/0x70 [45800.209749] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.217265] [] ? wake_up_atomic_t+0x30/0x30 [45800.223882] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.230980] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.238681] [] kthread+0xd1/0xe0 [45800.244151] [] ? insert_kthread_work+0x40/0x40 [45800.250983] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.258294] [] ? insert_kthread_work+0x40/0x40 [45800.265118] mdt_out01_013 S ffff99e869dd6300 0 7672 2 0x00000080 [45800.273060] Call Trace: [45800.275806] [] schedule+0x29/0x70 [45800.281432] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.288968] [] ? wake_up_atomic_t+0x30/0x30 [45800.295576] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.302646] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.310343] [] kthread+0xd1/0xe0 [45800.315810] [] ? insert_kthread_work+0x40/0x40 [45800.322624] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.329952] [] ? insert_kthread_work+0x40/0x40 [45800.336776] mdt_out01_014 S ffff99e869dd5280 0 7673 2 0x00000080 [45800.344735] Call Trace: [45800.347485] [] schedule+0x29/0x70 [45800.353124] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.360634] [] ? wake_up_atomic_t+0x30/0x30 [45800.367236] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.374319] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.382027] [] kthread+0xd1/0xe0 [45800.387489] [] ? insert_kthread_work+0x40/0x40 [45800.394314] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.401631] [] ? insert_kthread_work+0x40/0x40 [45800.408461] ldlm_cn01_019 S ffff99e869dd2100 0 7829 2 0x00000080 [45800.416433] Call Trace: [45800.419186] [] schedule+0x29/0x70 [45800.424819] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.432332] [] ? wake_up_atomic_t+0x30/0x30 [45800.438931] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.446044] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.453741] [] kthread+0xd1/0xe0 [45800.459210] [] ? insert_kthread_work+0x40/0x40 [45800.466047] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.473366] [] ? insert_kthread_work+0x40/0x40 [45800.480207] mdt_out01_015 S ffff99ec6ba96300 0 7834 2 0x00000080 [45800.488194] Call Trace: [45800.490967] [] schedule+0x29/0x70 [45800.496600] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.504110] [] ? wake_up_atomic_t+0x30/0x30 [45800.510713] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.517790] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.525492] [] kthread+0xd1/0xe0 [45800.530988] [] ? insert_kthread_work+0x40/0x40 [45800.537806] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.545123] [] ? insert_kthread_work+0x40/0x40 [45800.551975] mdt_out01_016 S ffff99ec42a81080 0 7993 2 0x00000080 [45800.559923] Call Trace: [45800.562674] [] schedule+0x29/0x70 [45800.568307] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.575815] [] ? wake_up_atomic_t+0x30/0x30 [45800.582410] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.589500] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.597210] [] kthread+0xd1/0xe0 [45800.602666] [] ? insert_kthread_work+0x40/0x40 [45800.609500] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.616817] [] ? insert_kthread_work+0x40/0x40 [45800.623641] ldlm_bl_07 S ffff99e869dd4200 0 7995 2 0x00000080 [45800.631596] Call Trace: [45800.634341] [] schedule+0x29/0x70 [45800.639985] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45800.647682] [] ? wake_up_atomic_t+0x30/0x30 [45800.654277] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45800.662546] [] kthread+0xd1/0xe0 [45800.668021] [] ? insert_kthread_work+0x40/0x40 [45800.674868] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.682177] [] ? insert_kthread_work+0x40/0x40 [45800.689002] ldlm_cn01_020 S ffff99e869dd3180 0 7996 2 0x00000080 [45800.696939] Call Trace: [45800.699687] [] schedule+0x29/0x70 [45800.705327] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45800.712838] [] ? wake_up_atomic_t+0x30/0x30 [45800.719443] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45800.726537] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45800.734238] [] kthread+0xd1/0xe0 [45800.739707] [] ? insert_kthread_work+0x40/0x40 [45800.746537] [] ret_from_fork_nospec_begin+0x21/0x21 [45800.753849] [] ? insert_kthread_work+0x40/0x40 [45800.760682] mdt00_014 S ffff99e84dd89080 0 8311 2 0x00000080 [45800.768619] Call Trace: [45800.771373] [] schedule+0x29/0x70 [45800.777015] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45800.784719] [] ? wake_up_atomic_t+0x30/0x30 [45800.791317] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45800.799249] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45800.807311] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45800.814786] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45800.823573] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45800.831014] [] osp_md_object_lock+0x160/0x300 [osp] [45800.838337] [] lod_object_lock+0xe2/0x7d0 [lod] [45800.845275] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45800.852931] [] ? lu_ucred+0x1e/0x30 [obdclass] [45800.859828] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45800.867941] [] mdd_object_lock+0x30/0xd0 [mdd] [45800.874785] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45800.882874] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45800.890591] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45800.898500] [] mdt_object_lock+0x20/0x30 [mdt] [45800.905355] [] mdt_create+0x6de/0xe40 [mdt] [45800.911968] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45800.919805] [] mdt_reint_create+0x3a0/0x460 [mdt] [45800.926942] [] mdt_reint_rec+0x8a/0x240 [mdt] [45800.933693] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45800.941053] [] mdt_reint+0x67/0x150 [mdt] [45800.947477] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45800.955262] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45800.963739] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45800.971708] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45800.980289] [] ? task_rq_unlock+0x20/0x20 [45800.986637] [] ? __wake_up+0x13/0x20 [45800.992549] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45800.999642] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.007344] [] kthread+0xd1/0xe0 [45801.012819] [] ? insert_kthread_work+0x40/0x40 [45801.019647] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.026959] [] ? insert_kthread_work+0x40/0x40 [45801.033784] mdt00_015 S ffff99ec6bbd3180 0 8312 2 0x00000080 [45801.041729] Call Trace: [45801.044491] [] schedule+0x29/0x70 [45801.050145] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.057640] [] ? wake_up_atomic_t+0x30/0x30 [45801.064243] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.071336] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.079038] [] kthread+0xd1/0xe0 [45801.084529] [] ? insert_kthread_work+0x40/0x40 [45801.091354] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.098665] [] ? insert_kthread_work+0x40/0x40 [45801.105505] mdt01_017 S ffff99ec6bbd5280 0 8314 2 0x00000080 [45801.113480] Call Trace: [45801.116241] [] schedule+0x29/0x70 [45801.121877] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45801.129647] [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] [45801.138028] [] ? wake_up_atomic_t+0x30/0x30 [45801.144618] [] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [45801.152652] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45801.161437] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45801.168764] [] mdt_object_local_lock+0x52f/0xba0 [mdt] [45801.176389] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45801.183775] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45801.192547] [] mdt_object_lock_internal+0x70/0x390 [mdt] [45801.200355] [] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [45801.208144] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [45801.215423] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [45801.223241] [] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [45801.230563] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [45801.237504] [] ? mdt_getattr_name+0x2c0/0x2c0 [mdt] [45801.244832] [] mdt_intent_policy+0x1a1/0x360 [mdt] [45801.252104] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [45801.259634] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [45801.267623] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [45801.274811] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [45801.282762] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [45801.290545] [] tgt_enqueue+0x64/0x240 [ptlrpc] [45801.297446] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45801.305234] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45801.313731] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45801.321700] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45801.330272] [] ? task_rq_unlock+0x20/0x20 [45801.336613] [] ? __wake_up+0x13/0x20 [45801.342527] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45801.349629] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.357332] [] kthread+0xd1/0xe0 [45801.362792] [] ? insert_kthread_work+0x40/0x40 [45801.369626] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.376943] [] ? insert_kthread_work+0x40/0x40 [45801.383776] mdt01_018 S ffff99ec6bbd2100 0 8315 2 0x00000080 [45801.391733] Call Trace: [45801.394487] [] schedule+0x29/0x70 [45801.400135] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.407629] [] ? wake_up_atomic_t+0x30/0x30 [45801.414224] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.421315] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.429042] [] kthread+0xd1/0xe0 [45801.434524] [] ? insert_kthread_work+0x40/0x40 [45801.441359] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.448670] [] ? insert_kthread_work+0x40/0x40 [45801.455514] mdt01_019 S ffff99ec6bbd6300 0 8316 2 0x00000080 [45801.463474] Call Trace: [45801.466226] [] schedule+0x29/0x70 [45801.471858] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.479370] [] ? wake_up_atomic_t+0x30/0x30 [45801.485957] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.493113] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.500823] [] kthread+0xd1/0xe0 [45801.506299] [] ? insert_kthread_work+0x40/0x40 [45801.513127] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.520439] [] ? insert_kthread_work+0x40/0x40 [45801.527262] ldlm_cb01_003 S ffff99e5b5ea4200 0 8780 2 0x00000080 [45801.535207] Call Trace: [45801.537946] [] schedule+0x29/0x70 [45801.543578] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.551098] [] ? wake_up_atomic_t+0x30/0x30 [45801.557694] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.564799] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.572494] [] kthread+0xd1/0xe0 [45801.577957] [] ? insert_kthread_work+0x40/0x40 [45801.584781] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.592099] [] ? insert_kthread_work+0x40/0x40 [45801.598923] mdt_io00_004 S ffff99e86a411080 0 9739 2 0x00000080 [45801.606874] Call Trace: [45801.609622] [] schedule+0x29/0x70 [45801.615257] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.622753] [] ? wake_up_atomic_t+0x30/0x30 [45801.629363] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.636454] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.644157] [] kthread+0xd1/0xe0 [45801.649625] [] ? insert_kthread_work+0x40/0x40 [45801.656453] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.663767] [] ? insert_kthread_work+0x40/0x40 [45801.670600] mdt_io01_005 S ffff99e5b63a2100 0 9896 2 0x00000080 [45801.678546] Call Trace: [45801.681300] [] schedule+0x29/0x70 [45801.686934] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.694444] [] ? wake_up_atomic_t+0x30/0x30 [45801.701048] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.708154] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.715867] [] kthread+0xd1/0xe0 [45801.721339] [] ? insert_kthread_work+0x40/0x40 [45801.728166] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.735476] [] ? insert_kthread_work+0x40/0x40 [45801.742297] mdt_io01_006 S ffff99e811a98000 0 9897 2 0x00000080 [45801.750256] Call Trace: [45801.753022] [] schedule+0x29/0x70 [45801.758645] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.766155] [] ? wake_up_atomic_t+0x30/0x30 [45801.772745] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.779835] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.787523] [] kthread+0xd1/0xe0 [45801.792993] [] ? insert_kthread_work+0x40/0x40 [45801.799819] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.807131] [] ? insert_kthread_work+0x40/0x40 [45801.813955] ldlm_cb01_004 S ffff99e811a99080 0 9899 2 0x00000080 [45801.821925] Call Trace: [45801.824687] [] schedule+0x29/0x70 [45801.830319] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.837828] [] ? wake_up_atomic_t+0x30/0x30 [45801.844430] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.851523] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.859249] [] kthread+0xd1/0xe0 [45801.864723] [] ? insert_kthread_work+0x40/0x40 [45801.871568] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.878879] [] ? insert_kthread_work+0x40/0x40 [45801.885703] ldlm_cb00_002 S ffff99e811a9a100 0 9901 2 0x00000080 [45801.893668] Call Trace: [45801.896423] [] schedule+0x29/0x70 [45801.902057] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.909560] [] ? wake_up_atomic_t+0x30/0x30 [45801.916176] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.923258] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45801.930960] [] kthread+0xd1/0xe0 [45801.936433] [] ? insert_kthread_work+0x40/0x40 [45801.943285] [] ret_from_fork_nospec_begin+0x21/0x21 [45801.950596] [] ? insert_kthread_work+0x40/0x40 [45801.957413] ldlm_cb00_003 S ffff99e86cf4e300 0 9903 2 0x00000080 [45801.965352] Call Trace: [45801.968117] [] schedule+0x29/0x70 [45801.973743] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45801.981252] [] ? wake_up_atomic_t+0x30/0x30 [45801.987847] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45801.994945] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.002642] [] kthread+0xd1/0xe0 [45802.008128] [] ? insert_kthread_work+0x40/0x40 [45802.014956] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.022262] [] ? insert_kthread_work+0x40/0x40 [45802.029085] ldlm_cb01_005 S ffff99e86cf4d280 0 9904 2 0x00000080 [45802.037051] Call Trace: [45802.039798] [] schedule+0x29/0x70 [45802.045432] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.052941] [] ? wake_up_atomic_t+0x30/0x30 [45802.059533] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.066631] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.074340] [] kthread+0xd1/0xe0 [45802.079815] [] ? insert_kthread_work+0x40/0x40 [45802.086642] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.093952] [] ? insert_kthread_work+0x40/0x40 [45802.100776] ldlm_cb00_004 S ffff99e86e71b180 0 9905 2 0x00000080 [45802.108727] Call Trace: [45802.111481] [] schedule+0x29/0x70 [45802.117128] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.124628] [] ? wake_up_atomic_t+0x30/0x30 [45802.131233] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.138331] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.146057] [] kthread+0xd1/0xe0 [45802.151525] [] ? insert_kthread_work+0x40/0x40 [45802.158352] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.165657] [] ? insert_kthread_work+0x40/0x40 [45802.172482] ldlm_cb01_006 S ffff99ec62ca5280 0 10177 2 0x00000080 [45802.180420] Call Trace: [45802.183174] [] schedule+0x29/0x70 [45802.188806] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.196323] [] ? wake_up_atomic_t+0x30/0x30 [45802.202933] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.210018] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.217736] [] kthread+0xd1/0xe0 [45802.223204] [] ? insert_kthread_work+0x40/0x40 [45802.230042] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.237331] [] ? insert_kthread_work+0x40/0x40 [45802.244169] kworker/29:2 S ffff99e8f597a100 0 12323 2 0x00000080 [45802.252121] Call Trace: [45802.254878] [] schedule+0x29/0x70 [45802.260453] [] worker_thread+0x1d9/0x3c0 [45802.266706] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45802.274049] [] kthread+0xd1/0xe0 [45802.279518] [] ? insert_kthread_work+0x40/0x40 [45802.286343] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.293654] [] ? insert_kthread_work+0x40/0x40 [45802.300476] mdt_out00_005 S ffff99e86a5a9080 0 12980 2 0x00000080 [45802.308401] Call Trace: [45802.311147] [] schedule+0x29/0x70 [45802.316779] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.324295] [] ? wake_up_atomic_t+0x30/0x30 [45802.330904] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.337991] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.345708] [] kthread+0xd1/0xe0 [45802.351177] [] ? insert_kthread_work+0x40/0x40 [45802.357997] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.365306] [] ? insert_kthread_work+0x40/0x40 [45802.372138] mdt_out00_006 S ffff99e86c17c200 0 12981 2 0x00000080 [45802.380073] Call Trace: [45802.382819] [] schedule+0x29/0x70 [45802.388463] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.395981] [] ? wake_up_atomic_t+0x30/0x30 [45802.402569] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.409660] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.417347] [] kthread+0xd1/0xe0 [45802.422809] [] ? insert_kthread_work+0x40/0x40 [45802.429636] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.436945] [] ? insert_kthread_work+0x40/0x40 [45802.443768] mdt_out00_007 S ffff99e86c17a100 0 12982 2 0x00000080 [45802.451719] Call Trace: [45802.454472] [] schedule+0x29/0x70 [45802.460129] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.467632] [] ? wake_up_atomic_t+0x30/0x30 [45802.474233] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.481326] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.489025] [] kthread+0xd1/0xe0 [45802.494507] [] ? insert_kthread_work+0x40/0x40 [45802.501335] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.508645] [] ? insert_kthread_work+0x40/0x40 [45802.515471] mdt_out00_008 S ffff99e86c17b180 0 12983 2 0x00000080 [45802.523432] Call Trace: [45802.526187] [] schedule+0x29/0x70 [45802.531826] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.539337] [] ? wake_up_atomic_t+0x30/0x30 [45802.545938] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.553018] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.560737] [] kthread+0xd1/0xe0 [45802.566200] [] ? insert_kthread_work+0x40/0x40 [45802.573027] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.580367] [] ? insert_kthread_work+0x40/0x40 [45802.587183] mdt_out00_009 S ffff99e86c178000 0 12988 2 0x00000080 [45802.595157] Call Trace: [45802.597911] [] schedule+0x29/0x70 [45802.603532] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.611058] [] ? wake_up_atomic_t+0x30/0x30 [45802.617647] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.624745] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.632442] [] kthread+0xd1/0xe0 [45802.637910] [] ? insert_kthread_work+0x40/0x40 [45802.644750] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.652087] [] ? insert_kthread_work+0x40/0x40 [45802.658918] mdt_out00_010 S ffff99e86c17e300 0 13022 2 0x00000080 [45802.666887] Call Trace: [45802.669646] [] schedule+0x29/0x70 [45802.675281] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.682791] [] ? wake_up_atomic_t+0x30/0x30 [45802.689379] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.696458] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.704153] [] kthread+0xd1/0xe0 [45802.709631] [] ? insert_kthread_work+0x40/0x40 [45802.716475] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.723787] [] ? insert_kthread_work+0x40/0x40 [45802.730617] mdt_out00_011 S ffff99e80fe5a100 0 13023 2 0x00000080 [45802.738558] Call Trace: [45802.741303] [] schedule+0x29/0x70 [45802.746935] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.754446] [] ? wake_up_atomic_t+0x30/0x30 [45802.761037] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.768161] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.775870] [] kthread+0xd1/0xe0 [45802.781349] [] ? insert_kthread_work+0x40/0x40 [45802.788175] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.795486] [] ? insert_kthread_work+0x40/0x40 [45802.802303] mdt_out00_012 S ffff99e84e7ce300 0 13062 2 0x00000080 [45802.810240] Call Trace: [45802.812987] [] schedule+0x29/0x70 [45802.818618] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.826139] [] ? wake_up_atomic_t+0x30/0x30 [45802.832746] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.839862] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.847564] [] kthread+0xd1/0xe0 [45802.853030] [] ? insert_kthread_work+0x40/0x40 [45802.859879] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.867187] [] ? insert_kthread_work+0x40/0x40 [45802.874035] ldlm_cb00_005 S ffff99ec07692100 0 13699 2 0x00000080 [45802.881980] Call Trace: [45802.884725] [] schedule+0x29/0x70 [45802.890357] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45802.897867] [] ? wake_up_atomic_t+0x30/0x30 [45802.904476] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45802.911576] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45802.919279] [] kthread+0xd1/0xe0 [45802.924747] [] ? insert_kthread_work+0x40/0x40 [45802.931580] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.938889] [] ? insert_kthread_work+0x40/0x40 [45802.945721] kworker/19:2 S ffff99e6b2008000 0 15739 2 0x00000080 [45802.953669] Call Trace: [45802.956416] [] schedule+0x29/0x70 [45802.961982] [] worker_thread+0x1d9/0x3c0 [45802.968234] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45802.975550] [] kthread+0xd1/0xe0 [45802.981021] [] ? insert_kthread_work+0x40/0x40 [45802.987847] [] ret_from_fork_nospec_begin+0x21/0x21 [45802.995167] [] ? insert_kthread_work+0x40/0x40 [45803.001990] mdt_io01_007 S ffff99e812769080 0 15900 2 0x00000080 [45803.009948] Call Trace: [45803.012684] [] schedule+0x29/0x70 [45803.018317] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.025827] [] ? wake_up_atomic_t+0x30/0x30 [45803.032414] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.039514] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.047204] [] kthread+0xd1/0xe0 [45803.052678] [] ? insert_kthread_work+0x40/0x40 [45803.059504] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.066813] [] ? insert_kthread_work+0x40/0x40 [45803.073638] ldlm_cn00_009 S ffff99e86c50a100 0 16656 2 0x00000080 [45803.081573] Call Trace: [45803.084320] [] schedule+0x29/0x70 [45803.089953] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.097456] [] ? wake_up_atomic_t+0x30/0x30 [45803.104053] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.111175] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.118878] [] kthread+0xd1/0xe0 [45803.124339] [] ? insert_kthread_work+0x40/0x40 [45803.131177] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.138488] [] ? insert_kthread_work+0x40/0x40 [45803.145312] ldlm_cn00_010 S ffff99e86cf49080 0 16657 2 0x00000080 [45803.153248] Call Trace: [45803.156001] [] schedule+0x29/0x70 [45803.161641] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.169172] [] ? wake_up_atomic_t+0x30/0x30 [45803.175773] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.182862] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.190563] [] kthread+0xd1/0xe0 [45803.196033] [] ? insert_kthread_work+0x40/0x40 [45803.202859] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.210177] [] ? insert_kthread_work+0x40/0x40 [45803.217001] ldlm_cn00_011 S ffff99e86cf4a100 0 16696 2 0x00000080 [45803.224979] Call Trace: [45803.227740] [] schedule+0x29/0x70 [45803.233378] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.240887] [] ? wake_up_atomic_t+0x30/0x30 [45803.247467] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.254552] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.262240] [] kthread+0xd1/0xe0 [45803.267708] [] ? insert_kthread_work+0x40/0x40 [45803.274533] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.281842] [] ? insert_kthread_work+0x40/0x40 [45803.288674] kworker/22:0 S ffff99e86cf4b180 0 16744 2 0x00000080 [45803.296650] Call Trace: [45803.299401] [] schedule+0x29/0x70 [45803.304967] [] worker_thread+0x1d9/0x3c0 [45803.311210] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45803.318513] [] kthread+0xd1/0xe0 [45803.323980] [] ? insert_kthread_work+0x40/0x40 [45803.330806] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.338116] [] ? insert_kthread_work+0x40/0x40 [45803.344939] ldlm_cn00_012 S ffff99e86a51a100 0 16973 2 0x00000080 [45803.352906] Call Trace: [45803.355659] [] schedule+0x29/0x70 [45803.361298] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.368806] [] ? wake_up_atomic_t+0x30/0x30 [45803.375408] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.382485] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.390180] [] kthread+0xd1/0xe0 [45803.395657] [] ? insert_kthread_work+0x40/0x40 [45803.402469] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.409778] [] ? insert_kthread_work+0x40/0x40 [45803.416610] ldlm_cn00_013 S ffff99e5b5336300 0 17013 2 0x00000080 [45803.424584] Call Trace: [45803.427341] [] schedule+0x29/0x70 [45803.432967] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.440479] [] ? wake_up_atomic_t+0x30/0x30 [45803.447063] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.454168] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.461863] [] kthread+0xd1/0xe0 [45803.467317] [] ? insert_kthread_work+0x40/0x40 [45803.474142] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.481470] [] ? insert_kthread_work+0x40/0x40 [45803.488301] ldlm_cn00_014 S ffff99e5b5334200 0 17023 2 0x00000080 [45803.496251] Call Trace: [45803.499005] [] schedule+0x29/0x70 [45803.504624] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.512118] [] ? wake_up_atomic_t+0x30/0x30 [45803.518711] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.525800] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.533501] [] kthread+0xd1/0xe0 [45803.538968] [] ? insert_kthread_work+0x40/0x40 [45803.545802] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.553137] [] ? insert_kthread_work+0x40/0x40 [45803.559962] mdt_rdpg00_009 S ffff99e81276b180 0 17955 2 0x00000080 [45803.567929] Call Trace: [45803.570682] [] schedule+0x29/0x70 [45803.576315] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.583830] [] ? wake_up_atomic_t+0x30/0x30 [45803.590417] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.597501] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.605211] [] kthread+0xd1/0xe0 [45803.610687] [] ? insert_kthread_work+0x40/0x40 [45803.617498] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.624809] [] ? insert_kthread_work+0x40/0x40 [45803.631635] mdt_rdpg00_010 S ffff99e86c3e2100 0 17957 2 0x00000080 [45803.639588] Call Trace: [45803.642327] [] schedule+0x29/0x70 [45803.647960] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.655469] [] ? wake_up_atomic_t+0x30/0x30 [45803.662043] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.669149] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.676880] [] kthread+0xd1/0xe0 [45803.682342] [] ? insert_kthread_work+0x40/0x40 [45803.689185] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.696495] [] ? insert_kthread_work+0x40/0x40 [45803.703320] ll_mgs_0003 S ffff99e86a5ae300 0 19566 2 0x00000080 [45803.711252] Call Trace: [45803.714005] [] schedule+0x29/0x70 [45803.719626] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.727126] [] ? wake_up_atomic_t+0x30/0x30 [45803.733727] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.740823] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.748526] [] kthread+0xd1/0xe0 [45803.753993] [] ? insert_kthread_work+0x40/0x40 [45803.760821] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.768139] [] ? insert_kthread_work+0x40/0x40 [45803.774954] ldlm_cn00_015 S ffff99e86a5ac200 0 19569 2 0x00000080 [45803.782891] Call Trace: [45803.785644] [] schedule+0x29/0x70 [45803.791277] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.798786] [] ? wake_up_atomic_t+0x30/0x30 [45803.805388] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.812471] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.820185] [] kthread+0xd1/0xe0 [45803.825654] [] ? insert_kthread_work+0x40/0x40 [45803.832482] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.839795] [] ? insert_kthread_work+0x40/0x40 [45803.846639] ldlm_cb01_007 S ffff99ec42a83180 0 22095 2 0x00000080 [45803.854610] Call Trace: [45803.857364] [] schedule+0x29/0x70 [45803.862999] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.870509] [] ? wake_up_atomic_t+0x30/0x30 [45803.877122] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.884218] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.891914] [] kthread+0xd1/0xe0 [45803.897368] [] ? insert_kthread_work+0x40/0x40 [45803.904201] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.911511] [] ? insert_kthread_work+0x40/0x40 [45803.918334] ldlm_cb01_008 S ffff99ec42a82100 0 22096 2 0x00000080 [45803.926278] Call Trace: [45803.929038] [] schedule+0x29/0x70 [45803.934659] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45803.942152] [] ? wake_up_atomic_t+0x30/0x30 [45803.948755] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45803.955845] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45803.963549] [] kthread+0xd1/0xe0 [45803.969019] [] ? insert_kthread_work+0x40/0x40 [45803.975846] [] ret_from_fork_nospec_begin+0x21/0x21 [45803.983156] [] ? insert_kthread_work+0x40/0x40 [45803.989993] ldlm_cb01_009 S ffff99ec42a85280 0 22097 2 0x00000080 [45803.997980] Call Trace: [45804.000723] [] schedule+0x29/0x70 [45804.006357] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45804.013873] [] ? wake_up_atomic_t+0x30/0x30 [45804.020475] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45804.027564] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45804.035266] [] kthread+0xd1/0xe0 [45804.040734] [] ? insert_kthread_work+0x40/0x40 [45804.047561] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.054873] [] ? insert_kthread_work+0x40/0x40 [45804.061722] ldlm_bl_08 S ffff99ec42a84200 0 22098 2 0x00000080 [45804.069694] Call Trace: [45804.072448] [] schedule+0x29/0x70 [45804.078071] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45804.085790] [] ? wake_up_atomic_t+0x30/0x30 [45804.092377] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45804.100660] [] kthread+0xd1/0xe0 [45804.106153] [] ? insert_kthread_work+0x40/0x40 [45804.112978] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.120296] [] ? insert_kthread_work+0x40/0x40 [45804.127140] ldlm_bl_09 S ffff99e86a410000 0 22099 2 0x00000080 [45804.135093] Call Trace: [45804.137846] [] schedule+0x29/0x70 [45804.143475] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45804.151192] [] ? wake_up_atomic_t+0x30/0x30 [45804.157793] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45804.166069] [] kthread+0xd1/0xe0 [45804.171538] [] ? insert_kthread_work+0x40/0x40 [45804.178366] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.185684] [] ? insert_kthread_work+0x40/0x40 [45804.192514] ldlm_bl_10 S ffff99e86a415280 0 22100 2 0x00000080 [45804.200449] Call Trace: [45804.203206] [] schedule+0x29/0x70 [45804.208830] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45804.216533] [] ? wake_up_atomic_t+0x30/0x30 [45804.223132] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45804.231418] [] kthread+0xd1/0xe0 [45804.236888] [] ? insert_kthread_work+0x40/0x40 [45804.243714] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.251031] [] ? insert_kthread_work+0x40/0x40 [45804.257862] ldlm_cb00_006 S ffff99e86a414200 0 22101 2 0x00000080 [45804.265806] Call Trace: [45804.268552] [] schedule+0x29/0x70 [45804.274191] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45804.281701] [] ? wake_up_atomic_t+0x30/0x30 [45804.288304] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45804.295396] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45804.303093] [] kthread+0xd1/0xe0 [45804.308569] [] ? insert_kthread_work+0x40/0x40 [45804.315404] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.322720] [] ? insert_kthread_work+0x40/0x40 [45804.329542] mdt00_016 S ffff99e5b547a100 0 22102 2 0x00000080 [45804.337495] Call Trace: [45804.340252] [] schedule+0x29/0x70 [45804.345885] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45804.353394] [] ? wake_up_atomic_t+0x30/0x30 [45804.359998] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45804.367070] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45804.374771] [] kthread+0xd1/0xe0 [45804.380246] [] ? insert_kthread_work+0x40/0x40 [45804.387066] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.394376] [] ? insert_kthread_work+0x40/0x40 [45804.401212] mdt00_017 S ffff99e81276c200 0 22104 2 0x00000080 [45804.409163] Call Trace: [45804.411917] [] schedule+0x29/0x70 [45804.417547] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45804.425250] [] ? wake_up_atomic_t+0x30/0x30 [45804.431840] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45804.439771] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45804.447846] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45804.455319] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45804.464105] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45804.471525] [] osp_md_object_lock+0x160/0x300 [osp] [45804.478850] [] lod_object_lock+0xe2/0x7d0 [lod] [45804.485790] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45804.493435] [] ? lu_ucred+0x1e/0x30 [obdclass] [45804.500289] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45804.508399] [] mdd_object_lock+0x30/0xd0 [mdd] [45804.515263] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45804.523353] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45804.531074] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45804.538983] [] mdt_object_lock+0x20/0x30 [mdt] [45804.545830] [] mdt_create+0x6de/0xe40 [mdt] [45804.552432] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45804.560261] [] mdt_reint_create+0x3a0/0x460 [mdt] [45804.567384] [] mdt_reint_rec+0x8a/0x240 [mdt] [45804.574145] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45804.581475] [] mdt_reint+0x67/0x150 [mdt] [45804.587904] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45804.595684] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45804.604164] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45804.612110] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45804.620702] [] ? wake_up_atomic_t+0x30/0x30 [45804.627299] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45804.634400] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45804.642095] [] kthread+0xd1/0xe0 [45804.647563] [] ? insert_kthread_work+0x40/0x40 [45804.654390] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.661706] [] ? insert_kthread_work+0x40/0x40 [45804.668529] mdt00_018 S ffff99e81276a100 0 22105 2 0x00000080 [45804.676465] Call Trace: [45804.679223] [] schedule+0x29/0x70 [45804.684853] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45804.692564] [] ? wake_up_atomic_t+0x30/0x30 [45804.699180] [] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [45804.707106] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [45804.715166] [] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [45804.722635] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45804.731420] [] ? mdt_obd_reconnect+0x210/0x210 [mdt] [45804.738847] [] osp_md_object_lock+0x160/0x300 [osp] [45804.746193] [] lod_object_lock+0xe2/0x7d0 [lod] [45804.753131] [] ? osp_md_index_lookup+0x25a/0x8c0 [osp] [45804.760808] [] ? lu_ucred+0x1e/0x30 [obdclass] [45804.767663] [] ? __mdd_permission_internal+0x39/0xb20 [mdd] [45804.775768] [] mdd_object_lock+0x30/0xd0 [mdd] [45804.782617] [] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [45804.790704] [] ? __mdd_lookup.isra.16+0x2a8/0x470 [mdd] [45804.798417] [] mdt_object_lock_internal+0x19c/0x390 [mdt] [45804.806324] [] mdt_object_lock+0x20/0x30 [mdt] [45804.813191] [] mdt_create+0x6de/0xe40 [mdt] [45804.819803] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45804.827630] [] mdt_reint_create+0x3a0/0x460 [mdt] [45804.834774] [] mdt_reint_rec+0x8a/0x240 [mdt] [45804.841530] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45804.848873] [] mdt_reint+0x67/0x150 [mdt] [45804.855304] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45804.863091] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45804.871579] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45804.879549] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45804.888150] [] ? wake_up_atomic_t+0x30/0x30 [45804.894756] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45804.901848] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45804.909552] [] kthread+0xd1/0xe0 [45804.915028] [] ? insert_kthread_work+0x40/0x40 [45804.921856] [] ret_from_fork_nospec_begin+0x21/0x21 [45804.929187] [] ? insert_kthread_work+0x40/0x40 [45804.936018] mdt01_020 D ffff99e81276e300 0 22106 2 0x00000080 [45804.943988] Call Trace: [45804.946734] [] schedule+0x29/0x70 [45804.952306] [] lbug_with_loc+0x7d/0xa0 [libcfs] [45804.959278] [] llog_osd_write_rec+0x1a3a/0x1a70 [obdclass] [45804.967283] [] llog_write_rec+0x293/0x590 [obdclass] [45804.974713] [] llog_cat_add_rec+0x1d9/0x980 [obdclass] [45804.982328] [] llog_add+0x182/0x1f0 [obdclass] [45804.989205] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [45804.997267] [] sub_updates_write+0x302/0xe3b [ptlrpc] [45805.004864] [] ? prepare_writing_updates.isra.10+0x139/0x14a [ptlrpc] [45805.014014] [] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [45805.021252] [] ? mutex_lock+0x12/0x2f [45805.027205] [] ? lod_attr_set+0x667/0xb40 [lod] [45805.034137] [] lod_trans_stop+0x25c/0x340 [lod] [45805.041103] [] mdd_trans_stop+0x2e/0x174 [mdd] [45805.047937] [] mdd_create+0x154a/0x1cd0 [mdd] [45805.054687] [] ? mdt_version_save+0x69/0xf0 [mdt] [45805.061828] [] mdo_create+0x46/0x48 [mdt] [45805.068228] [] mdt_create+0xab1/0xe40 [mdt] [45805.074840] [] ? req_capsule_get_size+0x30/0x70 [ptlrpc] [45805.082669] [] mdt_reint_create+0x3a0/0x460 [mdt] [45805.089805] [] mdt_reint_rec+0x8a/0x240 [mdt] [45805.096557] [] mdt_reint_internal+0x76c/0xb50 [mdt] [45805.103891] [] mdt_reint+0x67/0x150 [mdt] [45805.110324] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45805.118097] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45805.126574] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45805.134536] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45805.143112] [] ? wake_up_atomic_t+0x30/0x30 [45805.149714] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45805.156807] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.164510] [] kthread+0xd1/0xe0 [45805.169986] [] ? insert_kthread_work+0x40/0x40 [45805.176814] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.184127] [] ? insert_kthread_work+0x40/0x40 [45805.190959] mdt01_021 S ffff99e86fb2e300 0 22107 2 0x00000080 [45805.198904] Call Trace: [45805.201657] [] schedule+0x29/0x70 [45805.207285] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.214807] [] ? wake_up_atomic_t+0x30/0x30 [45805.221403] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.228494] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.236197] [] kthread+0xd1/0xe0 [45805.241672] [] ? insert_kthread_work+0x40/0x40 [45805.248499] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.255811] [] ? insert_kthread_work+0x40/0x40 [45805.262629] ldlm_cb00_007 S ffff99e86c2a9080 0 22183 2 0x00000080 [45805.270575] Call Trace: [45805.273330] [] schedule+0x29/0x70 [45805.278964] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.286472] [] ? wake_up_atomic_t+0x30/0x30 [45805.293066] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.300138] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.307847] [] kthread+0xd1/0xe0 [45805.313315] [] ? insert_kthread_work+0x40/0x40 [45805.320145] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.327456] [] ? insert_kthread_work+0x40/0x40 [45805.334286] mdt_out00_013 S ffff99ec07694200 0 23165 2 0x00000080 [45805.342237] Call Trace: [45805.345017] [] schedule+0x29/0x70 [45805.350638] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.358148] [] ? wake_up_atomic_t+0x30/0x30 [45805.364752] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.371843] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.379544] [] kthread+0xd1/0xe0 [45805.385010] [] ? insert_kthread_work+0x40/0x40 [45805.391840] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.399142] [] ? insert_kthread_work+0x40/0x40 [45805.405953] ldlm_cn00_016 S ffff99ec07695280 0 23207 2 0x00000080 [45805.413924] Call Trace: [45805.416675] [] schedule+0x29/0x70 [45805.422314] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.429816] [] ? wake_up_atomic_t+0x30/0x30 [45805.436417] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.443506] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.451219] [] kthread+0xd1/0xe0 [45805.456687] [] ? insert_kthread_work+0x40/0x40 [45805.463520] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.470830] [] ? insert_kthread_work+0x40/0x40 [45805.477662] mdt_out00_014 S ffff99ec07691080 0 23209 2 0x00000080 [45805.485620] Call Trace: [45805.488380] [] schedule+0x29/0x70 [45805.493992] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.501514] [] ? wake_up_atomic_t+0x30/0x30 [45805.508109] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.515203] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.522907] [] kthread+0xd1/0xe0 [45805.528382] [] ? insert_kthread_work+0x40/0x40 [45805.535219] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.542528] [] ? insert_kthread_work+0x40/0x40 [45805.549358] kworker/3:1 S ffff99e86d33a100 0 23848 2 0x00000080 [45805.557319] Call Trace: [45805.560074] [] schedule+0x29/0x70 [45805.565640] [] worker_thread+0x1d9/0x3c0 [45805.571876] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45805.579201] [] kthread+0xd1/0xe0 [45805.584669] [] ? insert_kthread_work+0x40/0x40 [45805.591497] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.598807] [] ? insert_kthread_work+0x40/0x40 [45805.605646] ldlm_cn00_017 S ffff99ec6d1e3180 0 24512 2 0x00000080 [45805.613580] Call Trace: [45805.616338] [] schedule+0x29/0x70 [45805.621963] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.629473] [] ? wake_up_atomic_t+0x30/0x30 [45805.636076] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.643155] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.650852] [] kthread+0xd1/0xe0 [45805.656329] [] ? insert_kthread_work+0x40/0x40 [45805.663160] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.670470] [] ? insert_kthread_work+0x40/0x40 [45805.677309] ldlm_cn00_018 S ffff99ec6d1e4200 0 24513 2 0x00000080 [45805.685279] Call Trace: [45805.688031] [] schedule+0x29/0x70 [45805.693653] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.701165] [] ? wake_up_atomic_t+0x30/0x30 [45805.707765] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.714857] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.722567] [] kthread+0xd1/0xe0 [45805.728042] [] ? insert_kthread_work+0x40/0x40 [45805.734867] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.742174] [] ? insert_kthread_work+0x40/0x40 [45805.749025] ldlm_cn00_019 S ffff99e5b5049080 0 24514 2 0x00000080 [45805.756944] Call Trace: [45805.759690] [] schedule+0x29/0x70 [45805.765333] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.772842] [] ? wake_up_atomic_t+0x30/0x30 [45805.779445] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.786544] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.794266] [] kthread+0xd1/0xe0 [45805.799735] [] ? insert_kthread_work+0x40/0x40 [45805.806562] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.813871] [] ? insert_kthread_work+0x40/0x40 [45805.820694] mdt_out00_015 S ffff99e811a9c200 0 24983 2 0x00000080 [45805.828646] Call Trace: [45805.831383] [] schedule+0x29/0x70 [45805.837002] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.844518] [] ? wake_up_atomic_t+0x30/0x30 [45805.851126] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.858243] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.866012] [] kthread+0xd1/0xe0 [45805.871484] [] ? insert_kthread_work+0x40/0x40 [45805.878295] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.885604] [] ? insert_kthread_work+0x40/0x40 [45805.892428] mdt_io01_008 S ffff99e811a9b180 0 25062 2 0x00000080 [45805.900374] Call Trace: [45805.903122] [] schedule+0x29/0x70 [45805.908758] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.916288] [] ? wake_up_atomic_t+0x30/0x30 [45805.922899] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45805.929989] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45805.937697] [] kthread+0xd1/0xe0 [45805.943160] [] ? insert_kthread_work+0x40/0x40 [45805.949979] [] ret_from_fork_nospec_begin+0x21/0x21 [45805.957296] [] ? insert_kthread_work+0x40/0x40 [45805.964120] mdt_out00_016 S ffff99e8382b9080 0 25139 2 0x00000080 [45805.972049] Call Trace: [45805.974803] [] schedule+0x29/0x70 [45805.980443] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45805.987960] [] ? wake_up_atomic_t+0x30/0x30 [45805.994563] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.001653] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.009354] [] kthread+0xd1/0xe0 [45806.014816] [] ? insert_kthread_work+0x40/0x40 [45806.021644] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.028953] [] ? insert_kthread_work+0x40/0x40 [45806.035777] kworker/20:2 S ffff99e810e53180 0 25456 2 0x00000080 [45806.043750] Call Trace: [45806.046503] [] schedule+0x29/0x70 [45806.052061] [] worker_thread+0x1d9/0x3c0 [45806.058306] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45806.065617] [] kthread+0xd1/0xe0 [45806.071079] [] ? insert_kthread_work+0x40/0x40 [45806.077904] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.085231] [] ? insert_kthread_work+0x40/0x40 [45806.092042] ldlm_cb00_008 S ffff99e7f4fc3180 0 25772 2 0x00000080 [45806.099980] Call Trace: [45806.102726] [] schedule+0x29/0x70 [45806.108377] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.115891] [] ? wake_up_atomic_t+0x30/0x30 [45806.122495] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.129586] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.137301] [] kthread+0xd1/0xe0 [45806.142770] [] ? insert_kthread_work+0x40/0x40 [45806.149595] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.156891] [] ? insert_kthread_work+0x40/0x40 [45806.163715] kworker/7:2 S ffff99e86c3e1080 0 29181 2 0x00000080 [45806.171678] Call Trace: [45806.174433] [] schedule+0x29/0x70 [45806.180007] [] worker_thread+0x1d9/0x3c0 [45806.186251] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45806.193560] [] kthread+0xd1/0xe0 [45806.199020] [] ? insert_kthread_work+0x40/0x40 [45806.205847] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.213158] [] ? insert_kthread_work+0x40/0x40 [45806.219975] mdt_io00_005 S ffff99e5b5e74200 0 30182 2 0x00000080 [45806.227917] Call Trace: [45806.230661] [] schedule+0x29/0x70 [45806.236321] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.243837] [] ? wake_up_atomic_t+0x30/0x30 [45806.250439] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.257530] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.265253] [] kthread+0xd1/0xe0 [45806.270715] [] ? insert_kthread_work+0x40/0x40 [45806.277541] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.284858] [] ? insert_kthread_work+0x40/0x40 [45806.291682] mdt_io00_006 S ffff99e80fe5d280 0 30183 2 0x00000080 [45806.299649] Call Trace: [45806.302396] [] schedule+0x29/0x70 [45806.308037] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.315547] [] ? wake_up_atomic_t+0x30/0x30 [45806.322140] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.329241] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.336929] [] kthread+0xd1/0xe0 [45806.342392] [] ? insert_kthread_work+0x40/0x40 [45806.349230] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.356539] [] ? insert_kthread_work+0x40/0x40 [45806.363378] mdt_rdpg01_012 S ffff99ec6b434200 0 30498 2 0x00000080 [45806.371356] Call Trace: [45806.374102] [] schedule+0x29/0x70 [45806.379721] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.387253] [] ? wake_up_atomic_t+0x30/0x30 [45806.393855] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.400946] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.408649] [] kthread+0xd1/0xe0 [45806.414116] [] ? insert_kthread_work+0x40/0x40 [45806.420950] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.428271] [] ? insert_kthread_work+0x40/0x40 [45806.435104] mdt_rdpg01_013 S ffff99e86cf48000 0 32883 2 0x00000080 [45806.443053] Call Trace: [45806.445807] [] schedule+0x29/0x70 [45806.451438] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.458952] [] ? wake_up_atomic_t+0x30/0x30 [45806.465555] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.472646] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.480347] [] kthread+0xd1/0xe0 [45806.485822] [] ? insert_kthread_work+0x40/0x40 [45806.492656] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.499974] [] ? insert_kthread_work+0x40/0x40 [45806.506791] ldlm_cn00_020 S ffff99e86c168000 0 33154 2 0x00000080 [45806.514742] Call Trace: [45806.517489] [] schedule+0x29/0x70 [45806.523129] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.530623] [] ? wake_up_atomic_t+0x30/0x30 [45806.537213] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.544337] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.552032] [] kthread+0xd1/0xe0 [45806.557509] [] ? insert_kthread_work+0x40/0x40 [45806.564336] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.571646] [] ? insert_kthread_work+0x40/0x40 [45806.578484] kworker/15:0 S ffff99ec4781d280 0 33360 2 0x00000080 [45806.586449] Call Trace: [45806.589213] [] schedule+0x29/0x70 [45806.594794] [] worker_thread+0x1d9/0x3c0 [45806.601039] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45806.608349] [] kthread+0xd1/0xe0 [45806.613825] [] ? insert_kthread_work+0x40/0x40 [45806.620661] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.627996] [] ? insert_kthread_work+0x40/0x40 [45806.634827] ldlm_cn01_021 S ffff99e84dd8a100 0 34143 2 0x00000080 [45806.642784] Call Trace: [45806.645536] [] schedule+0x29/0x70 [45806.651175] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.658687] [] ? wake_up_atomic_t+0x30/0x30 [45806.665281] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.672380] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.680090] [] kthread+0xd1/0xe0 [45806.685559] [] ? insert_kthread_work+0x40/0x40 [45806.692379] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.699700] [] ? insert_kthread_work+0x40/0x40 [45806.706531] ldlm_cn01_022 S ffff99e84dd8d280 0 34144 2 0x00000080 [45806.714465] Call Trace: [45806.717214] [] schedule+0x29/0x70 [45806.722853] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.730363] [] ? wake_up_atomic_t+0x30/0x30 [45806.736951] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.744048] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.751756] [] kthread+0xd1/0xe0 [45806.757253] [] ? insert_kthread_work+0x40/0x40 [45806.764080] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.771390] [] ? insert_kthread_work+0x40/0x40 [45806.778215] ldlm_cn01_023 S ffff99e84dd8e300 0 34145 2 0x00000080 [45806.786162] Call Trace: [45806.788908] [] schedule+0x29/0x70 [45806.794542] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.802037] [] ? wake_up_atomic_t+0x30/0x30 [45806.808646] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.815746] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.823448] [] kthread+0xd1/0xe0 [45806.828925] [] ? insert_kthread_work+0x40/0x40 [45806.835753] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.843063] [] ? insert_kthread_work+0x40/0x40 [45806.849887] ldlm_cn01_024 S ffff99e824238000 0 34147 2 0x00000080 [45806.857881] Call Trace: [45806.860636] [] schedule+0x29/0x70 [45806.866280] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.873796] [] ? wake_up_atomic_t+0x30/0x30 [45806.880413] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.887504] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.895209] [] kthread+0xd1/0xe0 [45806.900678] [] ? insert_kthread_work+0x40/0x40 [45806.907505] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.914816] [] ? insert_kthread_work+0x40/0x40 [45806.921639] ldlm_cn01_025 S ffff99e86b27a100 0 34148 2 0x00000080 [45806.929587] Call Trace: [45806.932338] [] schedule+0x29/0x70 [45806.937978] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45806.945498] [] ? wake_up_atomic_t+0x30/0x30 [45806.952086] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45806.959178] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45806.966865] [] kthread+0xd1/0xe0 [45806.972346] [] ? insert_kthread_work+0x40/0x40 [45806.979172] [] ret_from_fork_nospec_begin+0x21/0x21 [45806.986482] [] ? insert_kthread_work+0x40/0x40 [45806.993318] ldlm_cn01_026 S ffff99e86b27d280 0 34149 2 0x00000080 [45807.001297] Call Trace: [45807.004058] [] schedule+0x29/0x70 [45807.009699] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.017210] [] ? wake_up_atomic_t+0x30/0x30 [45807.023812] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.030897] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.038594] [] kthread+0xd1/0xe0 [45807.044061] [] ? insert_kthread_work+0x40/0x40 [45807.050888] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.058201] [] ? insert_kthread_work+0x40/0x40 [45807.065031] ldlm_cn01_027 S ffff99e86b27c200 0 34150 2 0x00000080 [45807.072995] Call Trace: [45807.075744] [] schedule+0x29/0x70 [45807.081389] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.088898] [] ? wake_up_atomic_t+0x30/0x30 [45807.095493] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.102577] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.110284] [] kthread+0xd1/0xe0 [45807.115753] [] ? insert_kthread_work+0x40/0x40 [45807.122578] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.129895] [] ? insert_kthread_work+0x40/0x40 [45807.136712] ldlm_cn01_028 S ffff99e5b54f5280 0 34151 2 0x00000080 [45807.144655] Call Trace: [45807.147409] [] schedule+0x29/0x70 [45807.153041] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.160550] [] ? wake_up_atomic_t+0x30/0x30 [45807.167145] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.174225] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.181923] [] kthread+0xd1/0xe0 [45807.187392] [] ? insert_kthread_work+0x40/0x40 [45807.194227] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.201544] [] ? insert_kthread_work+0x40/0x40 [45807.208371] ldlm_cn01_029 S ffff99e7dbfd0000 0 34265 2 0x00000080 [45807.216330] Call Trace: [45807.219075] [] schedule+0x29/0x70 [45807.224709] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.232223] [] ? wake_up_atomic_t+0x30/0x30 [45807.238818] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.245910] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.253620] [] kthread+0xd1/0xe0 [45807.259096] [] ? insert_kthread_work+0x40/0x40 [45807.265930] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.273243] [] ? insert_kthread_work+0x40/0x40 [45807.280079] mdt_out01_017 S ffff99e7dbfd1080 0 34266 2 0x00000080 [45807.288031] Call Trace: [45807.290783] [] schedule+0x29/0x70 [45807.296428] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.303939] [] ? wake_up_atomic_t+0x30/0x30 [45807.310534] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.317633] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.325353] [] kthread+0xd1/0xe0 [45807.330828] [] ? insert_kthread_work+0x40/0x40 [45807.337657] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.344959] [] ? insert_kthread_work+0x40/0x40 [45807.351776] ldlm_cn01_030 S ffff99e7dbfd2100 0 34267 2 0x00000080 [45807.359714] Call Trace: [45807.362458] [] schedule+0x29/0x70 [45807.368090] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.375598] [] ? wake_up_atomic_t+0x30/0x30 [45807.382207] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.389317] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.397018] [] kthread+0xd1/0xe0 [45807.402485] [] ? insert_kthread_work+0x40/0x40 [45807.409311] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.416614] [] ? insert_kthread_work+0x40/0x40 [45807.423430] mdt_out01_018 S ffff99e7dbfd3180 0 34307 2 0x00000080 [45807.431359] Call Trace: [45807.434104] [] schedule+0x29/0x70 [45807.439723] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.447234] [] ? wake_up_atomic_t+0x30/0x30 [45807.453842] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.460939] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.468644] [] kthread+0xd1/0xe0 [45807.474114] [] ? insert_kthread_work+0x40/0x40 [45807.480940] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.488254] [] ? insert_kthread_work+0x40/0x40 [45807.495094] ldlm_cn01_031 S ffff99e7dbfd4200 0 34308 2 0x00000080 [45807.503033] Call Trace: [45807.505787] [] schedule+0x29/0x70 [45807.511433] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.518942] [] ? wake_up_atomic_t+0x30/0x30 [45807.525544] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.532636] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.540343] [] kthread+0xd1/0xe0 [45807.545813] [] ? insert_kthread_work+0x40/0x40 [45807.552638] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.559948] [] ? insert_kthread_work+0x40/0x40 [45807.566771] kworker/u311:2 S ffff99e5b63a4200 0 34625 2 0x00000080 [45807.574721] Call Trace: [45807.577474] [] schedule+0x29/0x70 [45807.583038] [] worker_thread+0x1d9/0x3c0 [45807.589296] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45807.596594] [] kthread+0xd1/0xe0 [45807.602071] [] ? insert_kthread_work+0x40/0x40 [45807.608897] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.616208] [] ? insert_kthread_work+0x40/0x40 [45807.623025] kworker/4:2 S ffff99e86a5aa100 0 35102 2 0x00000080 [45807.630965] Call Trace: [45807.633718] [] schedule+0x29/0x70 [45807.639291] [] worker_thread+0x1d9/0x3c0 [45807.645541] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45807.652843] [] kthread+0xd1/0xe0 [45807.658311] [] ? insert_kthread_work+0x40/0x40 [45807.665138] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.672441] [] ? insert_kthread_work+0x40/0x40 [45807.679257] mdt01_022 S ffff99e86a5a8000 0 35178 2 0x00000080 [45807.687212] Call Trace: [45807.689957] [] schedule+0x29/0x70 [45807.695579] [] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [45807.703365] [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] [45807.711767] [] ? wake_up_atomic_t+0x30/0x30 [45807.718378] [] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [45807.726447] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45807.735225] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45807.742548] [] mdt_object_local_lock+0x52f/0xba0 [mdt] [45807.750163] [] ? mdt_object_alloc+0x320/0x320 [mdt] [45807.757523] [] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [45807.766325] [] mdt_object_lock_internal+0x70/0x390 [mdt] [45807.774139] [] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [45807.781901] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [45807.789186] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [45807.796991] [] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [45807.804333] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [45807.811312] [] ? mdt_getattr_name+0x2c0/0x2c0 [mdt] [45807.818639] [] mdt_intent_policy+0x1a1/0x360 [mdt] [45807.825919] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [45807.833444] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [45807.841450] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [45807.848633] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [45807.856591] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [45807.864454] [] tgt_enqueue+0x64/0x240 [ptlrpc] [45807.871363] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [45807.879138] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [45807.887621] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [45807.895592] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [45807.904174] [] ? task_rq_unlock+0x20/0x20 [45807.910513] [] ? __wake_up+0x13/0x20 [45807.916452] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [45807.923545] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45807.931241] [] kthread+0xd1/0xe0 [45807.936697] [] ? insert_kthread_work+0x40/0x40 [45807.943518] [] ret_from_fork_nospec_begin+0x21/0x21 [45807.950813] [] ? insert_kthread_work+0x40/0x40 [45807.957647] mdt01_023 S ffff99e86cf4c200 0 35179 2 0x00000080 [45807.965611] Call Trace: [45807.968385] [] schedule+0x29/0x70 [45807.974020] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45807.981523] [] ? wake_up_atomic_t+0x30/0x30 [45807.988128] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45807.995223] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.002924] [] kthread+0xd1/0xe0 [45808.008391] [] ? insert_kthread_work+0x40/0x40 [45808.015212] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.022515] [] ? insert_kthread_work+0x40/0x40 [45808.029344] mdt01_024 S ffff99e7bbed1080 0 35217 2 0x00000080 [45808.037318] Call Trace: [45808.040062] [] schedule+0x29/0x70 [45808.045695] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.053201] [] ? wake_up_atomic_t+0x30/0x30 [45808.059803] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.066881] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.074585] [] kthread+0xd1/0xe0 [45808.080055] [] ? insert_kthread_work+0x40/0x40 [45808.086881] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.094198] [] ? insert_kthread_work+0x40/0x40 [45808.101023] mdt01_025 S ffff99e5b5335280 0 35219 2 0x00000080 [45808.108976] Call Trace: [45808.111706] [] schedule+0x29/0x70 [45808.117352] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.124852] [] ? wake_up_atomic_t+0x30/0x30 [45808.131459] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.138551] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.146255] [] kthread+0xd1/0xe0 [45808.151725] [] ? insert_kthread_work+0x40/0x40 [45808.158552] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.165866] [] ? insert_kthread_work+0x40/0x40 [45808.172702] mdt01_026 S ffff99e5b5331080 0 35260 2 0x00000080 [45808.180649] Call Trace: [45808.183388] [] schedule+0x29/0x70 [45808.189022] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.196584] [] ? ptlrpc_at_remove_timed+0x4e/0xb0 [ptlrpc] [45808.204579] [] ? wake_up_atomic_t+0x30/0x30 [45808.211189] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.218288] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.225998] [] kthread+0xd1/0xe0 [45808.231467] [] ? insert_kthread_work+0x40/0x40 [45808.238317] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.245629] [] ? insert_kthread_work+0x40/0x40 [45808.252453] mdt01_027 S ffff99e5b5333180 0 35261 2 0x00000080 [45808.260387] Call Trace: [45808.263134] [] schedule+0x29/0x70 [45808.268753] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.276264] [] ? wake_up_atomic_t+0x30/0x30 [45808.282874] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.289973] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.297668] [] kthread+0xd1/0xe0 [45808.303138] [] ? insert_kthread_work+0x40/0x40 [45808.309957] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.317273] [] ? insert_kthread_work+0x40/0x40 [45808.324116] mdt01_028 S ffff99e5b5563180 0 35264 2 0x00000080 [45808.332063] Call Trace: [45808.334808] [] schedule+0x29/0x70 [45808.340460] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.347975] [] ? wake_up_atomic_t+0x30/0x30 [45808.354585] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.361668] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.369380] [] kthread+0xd1/0xe0 [45808.374843] [] ? insert_kthread_work+0x40/0x40 [45808.381657] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.388968] [] ? insert_kthread_work+0x40/0x40 [45808.395793] ldlm_cb00_009 S ffff99e84dc9a100 0 35423 2 0x00000080 [45808.403737] Call Trace: [45808.406483] [] schedule+0x29/0x70 [45808.412123] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.419639] [] ? wake_up_atomic_t+0x30/0x30 [45808.426247] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.433350] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.441052] [] kthread+0xd1/0xe0 [45808.446507] [] ? insert_kthread_work+0x40/0x40 [45808.453348] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.460660] [] ? insert_kthread_work+0x40/0x40 [45808.467483] ldlm_bl_11 S ffff99e5b5720000 0 35424 2 0x00000080 [45808.475455] Call Trace: [45808.478215] [] schedule+0x29/0x70 [45808.483840] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.491527] [] ? wake_up_atomic_t+0x30/0x30 [45808.498135] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.506422] [] kthread+0xd1/0xe0 [45808.511891] [] ? insert_kthread_work+0x40/0x40 [45808.518712] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.526015] [] ? insert_kthread_work+0x40/0x40 [45808.532852] mdt_fld_0002 S ffff99e7bbed0000 0 36795 2 0x00000080 [45808.540823] Call Trace: [45808.543562] [] schedule+0x29/0x70 [45808.549196] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45808.556690] [] ? wake_up_atomic_t+0x30/0x30 [45808.563283] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45808.570376] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45808.578092] [] kthread+0xd1/0xe0 [45808.583554] [] ? insert_kthread_work+0x40/0x40 [45808.590374] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.597703] [] ? insert_kthread_work+0x40/0x40 [45808.604534] ldlm_bl_12 S ffff99e867e74200 0 36865 2 0x00000080 [45808.612462] Call Trace: [45808.615208] [] schedule+0x29/0x70 [45808.620828] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.628525] [] ? wake_up_atomic_t+0x30/0x30 [45808.635128] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.643422] [] kthread+0xd1/0xe0 [45808.648889] [] ? insert_kthread_work+0x40/0x40 [45808.655714] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.663025] [] ? insert_kthread_work+0x40/0x40 [45808.669855] ldlm_bl_13 S ffff99e867e72100 0 36866 2 0x00000080 [45808.677811] Call Trace: [45808.680557] [] schedule+0x29/0x70 [45808.686186] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.693891] [] ? wake_up_atomic_t+0x30/0x30 [45808.700494] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.708781] [] kthread+0xd1/0xe0 [45808.714268] [] ? insert_kthread_work+0x40/0x40 [45808.721096] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.728425] [] ? insert_kthread_work+0x40/0x40 [45808.735248] ldlm_bl_14 S ffff99e867e75280 0 36867 2 0x00000080 [45808.743225] Call Trace: [45808.745978] [] schedule+0x29/0x70 [45808.751600] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.759296] [] ? wake_up_atomic_t+0x30/0x30 [45808.765896] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.774180] [] kthread+0xd1/0xe0 [45808.779649] [] ? insert_kthread_work+0x40/0x40 [45808.786468] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.793784] [] ? insert_kthread_work+0x40/0x40 [45808.800613] ldlm_bl_15 S ffff99e867e71080 0 36868 2 0x00000080 [45808.808564] Call Trace: [45808.811341] [] schedule+0x29/0x70 [45808.816969] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.824665] [] ? wake_up_atomic_t+0x30/0x30 [45808.831256] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.839534] [] kthread+0xd1/0xe0 [45808.845002] [] ? insert_kthread_work+0x40/0x40 [45808.851821] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.859138] [] ? insert_kthread_work+0x40/0x40 [45808.865970] ldlm_bl_16 S ffff99e867e73180 0 36869 2 0x00000080 [45808.873914] Call Trace: [45808.876660] [] schedule+0x29/0x70 [45808.882285] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.889992] [] ? wake_up_atomic_t+0x30/0x30 [45808.896578] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.904863] [] kthread+0xd1/0xe0 [45808.910357] [] ? insert_kthread_work+0x40/0x40 [45808.917198] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.924517] [] ? insert_kthread_work+0x40/0x40 [45808.931367] ldlm_bl_17 S ffff99e867e76300 0 36870 2 0x00000080 [45808.939303] Call Trace: [45808.942058] [] schedule+0x29/0x70 [45808.947691] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45808.955445] [] ? wake_up_atomic_t+0x30/0x30 [45808.962045] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45808.970358] [] kthread+0xd1/0xe0 [45808.975826] [] ? insert_kthread_work+0x40/0x40 [45808.982652] [] ret_from_fork_nospec_begin+0x21/0x21 [45808.989976] [] ? insert_kthread_work+0x40/0x40 [45808.996794] ldlm_bl_18 S ffff99e816432100 0 36871 2 0x00000080 [45809.004739] Call Trace: [45809.007492] [] schedule+0x29/0x70 [45809.013124] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45809.020840] [] ? wake_up_atomic_t+0x30/0x30 [45809.027437] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45809.035723] [] kthread+0xd1/0xe0 [45809.041192] [] ? insert_kthread_work+0x40/0x40 [45809.048029] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.055364] [] ? insert_kthread_work+0x40/0x40 [45809.062186] ldlm_bl_19 S ffff99e816433180 0 36872 2 0x00000080 [45809.070160] Call Trace: [45809.072913] [] schedule+0x29/0x70 [45809.078535] [] ldlm_bl_thread_main+0x76c/0xb10 [ptlrpc] [45809.086252] [] ? wake_up_atomic_t+0x30/0x30 [45809.092852] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] [45809.101142] [] kthread+0xd1/0xe0 [45809.106599] [] ? insert_kthread_work+0x40/0x40 [45809.113444] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.120761] [] ? insert_kthread_work+0x40/0x40 [45809.127591] kworker/12:0 S ffff99ec6b95e300 0 38864 2 0x00000080 [45809.135533] Call Trace: [45809.138286] [] schedule+0x29/0x70 [45809.143861] [] worker_thread+0x1d9/0x3c0 [45809.150105] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.157434] [] kthread+0xd1/0xe0 [45809.162904] [] ? insert_kthread_work+0x40/0x40 [45809.169734] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.177065] [] ? insert_kthread_work+0x40/0x40 [45809.183912] kworker/14:1 S ffff99e8f5414200 0 39374 2 0x00000080 [45809.191890] Call Trace: [45809.194651] [] schedule+0x29/0x70 [45809.200225] [] worker_thread+0x1d9/0x3c0 [45809.206454] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.213763] [] kthread+0xd1/0xe0 [45809.219244] [] ? insert_kthread_work+0x40/0x40 [45809.226070] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.233387] [] ? insert_kthread_work+0x40/0x40 [45809.240222] mdt_io01_009 S ffff99e7bb5c6300 0 42596 2 0x00000080 [45809.248208] Call Trace: [45809.250980] [] schedule+0x29/0x70 [45809.256609] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45809.264125] [] ? wake_up_atomic_t+0x30/0x30 [45809.270724] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45809.277822] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45809.285528] [] kthread+0xd1/0xe0 [45809.291013] [] ? insert_kthread_work+0x40/0x40 [45809.297843] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.305174] [] ? insert_kthread_work+0x40/0x40 [45809.312024] mdt_io01_010 S ffff99ec3e558000 0 42597 2 0x00000080 [45809.320029] Call Trace: [45809.322805] [] schedule+0x29/0x70 [45809.328453] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45809.335969] [] ? wake_up_atomic_t+0x30/0x30 [45809.342566] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45809.349658] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45809.357385] [] kthread+0xd1/0xe0 [45809.362862] [] ? insert_kthread_work+0x40/0x40 [45809.369697] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.377013] [] ? insert_kthread_work+0x40/0x40 [45809.383859] mdt_io00_007 S ffff99ec3e559080 0 42598 2 0x00000080 [45809.391818] Call Trace: [45809.394578] [] schedule+0x29/0x70 [45809.400211] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45809.407722] [] ? wake_up_atomic_t+0x30/0x30 [45809.414315] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45809.421453] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45809.429141] [] kthread+0xd1/0xe0 [45809.434609] [] ? insert_kthread_work+0x40/0x40 [45809.441438] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.448755] [] ? insert_kthread_work+0x40/0x40 [45809.455578] kworker/u305:0 S ffff99e86d33e300 0 42955 2 0x00000080 [45809.463519] Call Trace: [45809.466272] [] schedule+0x29/0x70 [45809.471838] [] worker_thread+0x1d9/0x3c0 [45809.478091] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.485399] [] kthread+0xd1/0xe0 [45809.490875] [] ? insert_kthread_work+0x40/0x40 [45809.497709] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.505027] [] ? insert_kthread_work+0x40/0x40 [45809.511858] mdt_seqm_0002 S ffff99ec6941e300 0 43901 2 0x00000080 [45809.519815] Call Trace: [45809.522568] [] schedule+0x29/0x70 [45809.528209] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45809.535712] [] ? wake_up_atomic_t+0x30/0x30 [45809.542305] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45809.549428] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45809.557137] [] kthread+0xd1/0xe0 [45809.562612] [] ? system_call_after_swapgs+0x96/0x13a [45809.570027] [] ? insert_kthread_work+0x40/0x40 [45809.576855] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.584174] [] ? insert_kthread_work+0x40/0x40 [45809.591005] kworker/13:1 S ffff99ec3e55b180 0 44014 2 0x00000080 [45809.598976] Call Trace: [45809.601731] [] schedule+0x29/0x70 [45809.607289] [] worker_thread+0x1d9/0x3c0 [45809.613534] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.620850] [] kthread+0xd1/0xe0 [45809.626316] [] ? insert_kthread_work+0x40/0x40 [45809.633141] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.640473] [] ? insert_kthread_work+0x40/0x40 [45809.647297] kworker/1:0 S ffff99e86b14e300 0 44368 2 0x00000080 [45809.655256] Call Trace: [45809.658000] [] schedule+0x29/0x70 [45809.663558] [] worker_thread+0x1d9/0x3c0 [45809.669807] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.677116] [] kthread+0xd1/0xe0 [45809.682584] [] ? insert_kthread_work+0x40/0x40 [45809.689416] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.696713] [] ? insert_kthread_work+0x40/0x40 [45809.703544] mdt_fld_0003 S ffff99eb653f1080 0 44997 2 0x00000080 [45809.711496] Call Trace: [45809.714249] [] schedule+0x29/0x70 [45809.719873] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45809.727400] [] ? wake_up_atomic_t+0x30/0x30 [45809.734010] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45809.741120] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45809.748820] [] kthread+0xd1/0xe0 [45809.754295] [] ? insert_kthread_work+0x40/0x40 [45809.761128] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.768454] [] ? insert_kthread_work+0x40/0x40 [45809.775293] kworker/14:2 S ffff99ec07613180 0 47385 2 0x00000080 [45809.783245] Call Trace: [45809.785997] [] schedule+0x29/0x70 [45809.791557] [] worker_thread+0x1d9/0x3c0 [45809.797804] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.805113] [] kthread+0xd1/0xe0 [45809.810584] [] ? insert_kthread_work+0x40/0x40 [45809.817420] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.824737] [] ? insert_kthread_work+0x40/0x40 [45809.831567] kworker/24:2 S ffff99ec6a410000 0 47701 2 0x00000080 [45809.839541] Call Trace: [45809.842302] [] schedule+0x29/0x70 [45809.847867] [] worker_thread+0x1d9/0x3c0 [45809.854112] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.861409] [] kthread+0xd1/0xe0 [45809.866887] [] ? insert_kthread_work+0x40/0x40 [45809.873714] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.881046] [] ? insert_kthread_work+0x40/0x40 [45809.887875] kworker/21:0 S ffff99e7f4fc1080 0 48578 2 0x00000080 [45809.895830] Call Trace: [45809.898577] [] schedule+0x29/0x70 [45809.904148] [] worker_thread+0x1d9/0x3c0 [45809.910413] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.917723] [] kthread+0xd1/0xe0 [45809.923192] [] ? insert_kthread_work+0x40/0x40 [45809.930020] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.937332] [] ? insert_kthread_work+0x40/0x40 [45809.944161] kworker/23:2 S ffff99e86c2ad280 0 53244 2 0x00000080 [45809.952177] Call Trace: [45809.954940] [] schedule+0x29/0x70 [45809.960513] [] worker_thread+0x1d9/0x3c0 [45809.966758] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45809.974070] [] kthread+0xd1/0xe0 [45809.979538] [] ? insert_kthread_work+0x40/0x40 [45809.986352] [] ret_from_fork_nospec_begin+0x21/0x21 [45809.993679] [] ? insert_kthread_work+0x40/0x40 [45810.000511] kworker/11:0 S ffff99e8f5594200 0 55268 2 0x00000080 [45810.008451] Call Trace: [45810.011222] [] schedule+0x29/0x70 [45810.016793] [] worker_thread+0x1d9/0x3c0 [45810.023037] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.030349] [] kthread+0xd1/0xe0 [45810.035817] [] ? insert_kthread_work+0x40/0x40 [45810.042641] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.049966] [] ? insert_kthread_work+0x40/0x40 [45810.056803] kworker/15:1 S ffff99e8f5591080 0 55270 2 0x00000080 [45810.064775] Call Trace: [45810.067530] [] schedule+0x29/0x70 [45810.073102] [] worker_thread+0x1d9/0x3c0 [45810.079348] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.086652] [] kthread+0xd1/0xe0 [45810.092128] [] ? insert_kthread_work+0x40/0x40 [45810.098960] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.106271] [] ? insert_kthread_work+0x40/0x40 [45810.113102] kworker/5:2 S ffff99e7f4fc2100 0 56099 2 0x00000080 [45810.121043] Call Trace: [45810.123790] [] schedule+0x29/0x70 [45810.129357] [] worker_thread+0x1d9/0x3c0 [45810.135601] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.142917] [] kthread+0xd1/0xe0 [45810.148404] [] ? insert_kthread_work+0x40/0x40 [45810.155228] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.162539] [] ? insert_kthread_work+0x40/0x40 [45810.169363] kworker/4:1 S ffff99e86c2aa100 0 56137 2 0x00000080 [45810.177356] Call Trace: [45810.180109] [] schedule+0x29/0x70 [45810.185676] [] worker_thread+0x1d9/0x3c0 [45810.191930] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.199249] [] kthread+0xd1/0xe0 [45810.204723] [] ? insert_kthread_work+0x40/0x40 [45810.211550] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.218861] [] ? insert_kthread_work+0x40/0x40 [45810.225685] kworker/2:0 S ffff99e84e7c6300 0 56139 2 0x00000080 [45810.233623] Call Trace: [45810.236362] [] schedule+0x29/0x70 [45810.241919] [] worker_thread+0x1d9/0x3c0 [45810.248163] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.255469] [] kthread+0xd1/0xe0 [45810.260950] [] ? insert_kthread_work+0x40/0x40 [45810.267760] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.275076] [] ? insert_kthread_work+0x40/0x40 [45810.281913] kworker/31:0 S ffff99ec69698000 0 56178 2 0x00000080 [45810.289880] Call Trace: [45810.292633] [] schedule+0x29/0x70 [45810.298208] [] worker_thread+0x1d9/0x3c0 [45810.304471] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.311786] [] kthread+0xd1/0xe0 [45810.317286] [] ? insert_kthread_work+0x40/0x40 [45810.324118] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.331431] [] ? insert_kthread_work+0x40/0x40 [45810.338269] kworker/0:0 S ffff99e5b55b1080 0 59042 2 0x00000080 [45810.346229] Call Trace: [45810.348974] [] schedule+0x29/0x70 [45810.354540] [] worker_thread+0x1d9/0x3c0 [45810.360784] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.368101] [] kthread+0xd1/0xe0 [45810.373571] [] ? insert_kthread_work+0x40/0x40 [45810.380399] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.387703] [] ? insert_kthread_work+0x40/0x40 [45810.394520] kworker/1:1 S ffff99e816435280 0 59353 2 0x00000080 [45810.402474] Call Trace: [45810.405227] [] schedule+0x29/0x70 [45810.410791] [] worker_thread+0x1d9/0x3c0 [45810.417036] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.424346] [] kthread+0xd1/0xe0 [45810.429816] [] ? insert_kthread_work+0x40/0x40 [45810.436628] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.443938] [] ? insert_kthread_work+0x40/0x40 [45810.450763] kworker/17:2 S ffff99e810e56300 0 59730 2 0x00000080 [45810.458736] Call Trace: [45810.461485] [] schedule+0x29/0x70 [45810.467065] [] worker_thread+0x1d9/0x3c0 [45810.473310] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.480620] [] kthread+0xd1/0xe0 [45810.486090] [] ? insert_kthread_work+0x40/0x40 [45810.492916] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.500225] [] ? insert_kthread_work+0x40/0x40 [45810.507044] kworker/u311:1 S ffff99e86d33c200 0 60143 2 0x00000080 [45810.515004] Call Trace: [45810.517750] [] schedule+0x29/0x70 [45810.523328] [] worker_thread+0x1d9/0x3c0 [45810.529572] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.536890] [] kthread+0xd1/0xe0 [45810.542359] [] ? insert_kthread_work+0x40/0x40 [45810.549184] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.556490] [] ? insert_kthread_work+0x40/0x40 [45810.563311] kworker/21:1 S ffff99e812768000 0 61122 2 0x00000080 [45810.571227] Call Trace: [45810.573980] [] schedule+0x29/0x70 [45810.579545] [] worker_thread+0x1d9/0x3c0 [45810.585799] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.593118] [] kthread+0xd1/0xe0 [45810.598587] [] ? insert_kthread_work+0x40/0x40 [45810.605426] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.612729] [] ? insert_kthread_work+0x40/0x40 [45810.619547] kworker/8:1 S ffff99ec6941c200 0 62687 2 0x00000080 [45810.627497] Call Trace: [45810.630259] [] schedule+0x29/0x70 [45810.635826] [] worker_thread+0x1d9/0x3c0 [45810.642078] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.649392] [] kthread+0xd1/0xe0 [45810.654887] [] ? insert_kthread_work+0x40/0x40 [45810.661698] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.669017] [] ? insert_kthread_work+0x40/0x40 [45810.675847] kworker/9:0 S ffff99e8f54cd280 0 62761 2 0x00000080 [45810.683777] Call Trace: [45810.686524] [] schedule+0x29/0x70 [45810.692097] [] worker_thread+0x1d9/0x3c0 [45810.698350] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.705647] [] kthread+0xd1/0xe0 [45810.711108] [] ? insert_kthread_work+0x40/0x40 [45810.717944] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.725262] [] ? insert_kthread_work+0x40/0x40 [45810.732085] kworker/u306:1 S ffff99e8f5aaa100 0 62810 2 0x00000080 [45810.740035] Call Trace: [45810.742781] [] schedule+0x29/0x70 [45810.748345] [] worker_thread+0x1d9/0x3c0 [45810.754589] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.761905] [] kthread+0xd1/0xe0 [45810.767367] [] ? insert_kthread_work+0x40/0x40 [45810.774201] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.781506] [] ? insert_kthread_work+0x40/0x40 [45810.788350] kworker/10:2 S ffff99e8f59d1080 0 63910 2 0x00000080 [45810.796316] Call Trace: [45810.799079] [] schedule+0x29/0x70 [45810.804637] [] worker_thread+0x1d9/0x3c0 [45810.810890] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.818208] [] kthread+0xd1/0xe0 [45810.823671] [] ? insert_kthread_work+0x40/0x40 [45810.830501] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.837807] [] ? insert_kthread_work+0x40/0x40 [45810.844638] kworker/u306:2 S ffff99ec6ac22100 0 64293 2 0x00000080 [45810.852604] Call Trace: [45810.855354] [] schedule+0x29/0x70 [45810.860921] [] worker_thread+0x1d9/0x3c0 [45810.867166] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.874470] [] kthread+0xd1/0xe0 [45810.879958] [] ? insert_kthread_work+0x40/0x40 [45810.886776] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.894084] [] ? insert_kthread_work+0x40/0x40 [45810.900930] kworker/8:0 S ffff99e8f59d6300 0 64491 2 0x00000080 [45810.908904] Call Trace: [45810.911648] [] schedule+0x29/0x70 [45810.917222] [] worker_thread+0x1d9/0x3c0 [45810.923464] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.930766] [] kthread+0xd1/0xe0 [45810.936223] [] ? insert_kthread_work+0x40/0x40 [45810.943034] [] ret_from_fork_nospec_begin+0x21/0x21 [45810.950358] [] ? insert_kthread_work+0x40/0x40 [45810.957211] kworker/6:0 S ffff99e86c17d280 0 64532 2 0x00000080 [45810.965184] Call Trace: [45810.967946] [] schedule+0x29/0x70 [45810.973531] [] worker_thread+0x1d9/0x3c0 [45810.979783] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45810.987101] [] kthread+0xd1/0xe0 [45810.992570] [] ? insert_kthread_work+0x40/0x40 [45810.999389] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.006693] [] ? insert_kthread_work+0x40/0x40 [45811.013530] mdt_io01_011 S ffff99eb0559c200 0 64774 2 0x00000080 [45811.021496] Call Trace: [45811.024259] [] schedule+0x29/0x70 [45811.029885] [] ptlrpc_wait_event+0x4c3/0x5c0 [ptlrpc] [45811.037412] [] ? wake_up_atomic_t+0x30/0x30 [45811.044046] [] ptlrpc_main+0xacd/0x15e0 [ptlrpc] [45811.051148] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [45811.058851] [] kthread+0xd1/0xe0 [45811.064319] [] ? insert_kthread_work+0x40/0x40 [45811.071132] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.078461] [] ? insert_kthread_work+0x40/0x40 [45811.085294] kworker/0:2 S ffff99e840e7d280 0 67752 2 0x00000080 [45811.093279] Call Trace: [45811.096041] [] schedule+0x29/0x70 [45811.101614] [] worker_thread+0x1d9/0x3c0 [45811.107846] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.115155] [] kthread+0xd1/0xe0 [45811.120623] [] ? insert_kthread_work+0x40/0x40 [45811.127461] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.134764] [] ? insert_kthread_work+0x40/0x40 [45811.141578] kworker/3:2 R running task 0 68185 2 0x00000080 [45811.149512] Call Trace: [45811.152264] [] schedule+0x29/0x70 [45811.157836] [] worker_thread+0x1d9/0x3c0 [45811.164094] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.171398] [] kthread+0xd1/0xe0 [45811.176859] [] ? system_call_after_swapgs+0x96/0x13a [45811.184275] [] ? insert_kthread_work+0x40/0x40 [45811.191088] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.198399] [] ? insert_kthread_work+0x40/0x40 [45811.205222] kworker/u312:1 S ffff99ec6b5a2100 0 68506 2 0x00000080 [45811.213200] Call Trace: [45811.215957] [] schedule+0x29/0x70 [45811.221519] [] worker_thread+0x1d9/0x3c0 [45811.227770] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.235095] [] kthread+0xd1/0xe0 [45811.240566] [] ? insert_kthread_work+0x40/0x40 [45811.247395] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.254700] [] ? insert_kthread_work+0x40/0x40 [45811.261533] kworker/12:1 S ffff99ec6b5a0000 0 68681 2 0x00000080 [45811.269509] Call Trace: [45811.272271] [] schedule+0x29/0x70 [45811.277837] [] worker_thread+0x1d9/0x3c0 [45811.284088] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.291385] [] kthread+0xd1/0xe0 [45811.296860] [] ? insert_kthread_work+0x40/0x40 [45811.303698] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.311025] [] ? insert_kthread_work+0x40/0x40 [45811.317877] kworker/13:2 S ffff99ec6ba73180 0 70510 2 0x00000080 [45811.325827] Call Trace: [45811.328573] [] schedule+0x29/0x70 [45811.334145] [] worker_thread+0x1d9/0x3c0 [45811.340393] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.347703] [] kthread+0xd1/0xe0 [45811.353187] [] ? insert_kthread_work+0x40/0x40 [45811.360027] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.367338] [] ? insert_kthread_work+0x40/0x40 [45811.374163] kworker/u304:2 S ffff99e816434200 0 71184 2 0x00000080 [45811.382128] Call Trace: [45811.384870] [] schedule+0x29/0x70 [45811.390458] [] worker_thread+0x1d9/0x3c0 [45811.396688] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.403990] [] kthread+0xd1/0xe0 [45811.409477] [] ? insert_kthread_work+0x40/0x40 [45811.416297] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.423608] [] ? insert_kthread_work+0x40/0x40 [45811.430452] kworker/u304:0 S ffff99e816431080 0 71789 2 0x00000080 [45811.438401] Call Trace: [45811.441147] [] schedule+0x29/0x70 [45811.446705] [] worker_thread+0x1d9/0x3c0 [45811.452956] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.460275] [] kthread+0xd1/0xe0 [45811.465745] [] ? insert_kthread_work+0x40/0x40 [45811.472573] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.479877] [] ? insert_kthread_work+0x40/0x40 [45811.486698] pickup S ffff99eb653f2100 0 72921 1925 0x00000080 [45811.494651] Call Trace: [45811.497406] [] schedule+0x29/0x70 [45811.502980] [] schedule_hrtimeout_range_clock+0xb2/0x150 [45811.510767] [] ? hrtimer_get_res+0x50/0x50 [45811.517212] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [45811.525208] [] schedule_hrtimeout_range+0x13/0x20 [45811.532333] [] ep_poll+0x23e/0x360 [45811.537997] [] ? wake_up_state+0x20/0x20 [45811.544248] [] SyS_epoll_wait+0xed/0x120 [45811.550497] [] ? system_call_after_swapgs+0x96/0x13a [45811.557903] [] system_call_fastpath+0x25/0x2a [45811.564626] [] ? system_call_after_swapgs+0xa2/0x13a [45811.572029] kworker/18:0 S ffff99e7dbfd5280 0 74257 2 0x00000080 [45811.580007] Call Trace: [45811.582755] [] schedule+0x29/0x70 [45811.588327] [] worker_thread+0x1d9/0x3c0 [45811.594571] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.601881] [] kthread+0xd1/0xe0 [45811.607349] [] ? insert_kthread_work+0x40/0x40 [45811.614180] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.621483] [] ? insert_kthread_work+0x40/0x40 [45811.628326] kworker/25:2 S ffff99e8f5ba1080 0 76446 2 0x00000080 [45811.636301] Call Trace: [45811.639056] [] schedule+0x29/0x70 [45811.644622] [] worker_thread+0x1d9/0x3c0 [45811.650866] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.658182] [] kthread+0xd1/0xe0 [45811.663646] [] ? insert_kthread_work+0x40/0x40 [45811.670499] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.677815] [] ? insert_kthread_work+0x40/0x40 [45811.684638] kworker/u304:1 S ffff99ec6bd49080 0 77030 2 0x00000080 [45811.692581] Call Trace: [45811.695334] [] schedule+0x29/0x70 [45811.700900] [] worker_thread+0x1d9/0x3c0 [45811.707151] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.714482] [] kthread+0xd1/0xe0 [45811.719956] [] ? insert_kthread_work+0x40/0x40 [45811.726779] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.734089] [] ? insert_kthread_work+0x40/0x40 [45811.740913] kworker/26:0 S ffff99ec6bd48000 0 77562 2 0x00000080 [45811.748870] Call Trace: [45811.751615] [] schedule+0x29/0x70 [45811.757190] [] worker_thread+0x1d9/0x3c0 [45811.763438] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.770747] [] kthread+0xd1/0xe0 [45811.776207] [] ? insert_kthread_work+0x40/0x40 [45811.783035] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.790345] [] ? insert_kthread_work+0x40/0x40 [45811.797190] kworker/u312:2 S ffff99e8f5b53180 0 78186 2 0x00000080 [45811.805171] Call Trace: [45811.807923] [] schedule+0x29/0x70 [45811.813493] [] worker_thread+0x1d9/0x3c0 [45811.819728] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.827031] [] kthread+0xd1/0xe0 [45811.832495] [] ? insert_kthread_work+0x40/0x40 [45811.839320] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.846631] [] ? insert_kthread_work+0x40/0x40 [45811.853485] kworker/18:2 S ffff99e86c3e0000 0 78505 2 0x00000080 [45811.861436] Call Trace: [45811.864187] [] schedule+0x29/0x70 [45811.869758] [] worker_thread+0x1d9/0x3c0 [45811.876018] [] ? manage_workers.isra.26+0x2a0/0x2a0 [45811.883345] [] kthread+0xd1/0xe0 [45811.888814] [] ? insert_kthread_work+0x40/0x40 [45811.895629] [] ret_from_fork_nospec_begin+0x21/0x21 [45811.902941] [] ? insert_kthread_work+0x40/0x40 [45811.909764] sshd D ffff99e7a37de300 0 78771 1801 0x00000080 [45811.917717] Call Trace: [45811.920476] [] schedule+0x29/0x70 [45811.926038] [] schedule_timeout+0x221/0x2d0 [45811.932581] [] ? sched_clock+0x9/0x10 [45811.938554] [] ? sched_clock_cpu+0x85/0xc0 [45811.944998] [] ? put_prev_entity+0x31/0x400 [45811.951537] [] ? try_to_wake_up+0x192/0x390 [45811.958069] [] wait_for_completion+0xfd/0x140 [45811.964798] [] ? wake_up_state+0x20/0x20 [45811.971035] [] flush_work+0x10a/0x1b0 [45811.976998] [] ? move_linked_works+0x90/0x90 [45811.983624] [] tty_flush_to_ldisc+0x15/0x20 [45811.990175] [] n_tty_poll+0x6d/0x1d0 [45811.996023] [] tty_poll+0x75/0x90 [45812.001597] [] do_select+0x3a8/0x7c0 [45812.007473] [] ? get_target_pstate_use_performance+0x8f/0xc0 [45812.015654] [] ? __pollwait+0xf0/0xf0 [45812.021606] [] ? __pollwait+0xf0/0xf0 [45812.027571] [] ? __pollwait+0xf0/0xf0 [45812.033514] [] ? __pollwait+0xf0/0xf0 [45812.039483] [] ? tracing_is_on+0x15/0x30 [45812.045713] [] ? tracing_record_cmdline+0x1d/0x120 [45812.052927] [] ? probe_sched_wakeup+0x2b/0xa0 [45812.059654] [] ? ttwu_do_wakeup+0xb5/0xe0 [45812.065994] [] ? try_to_wake_up+0x192/0x390 [45812.072527] [] ? default_wake_function+0x12/0x20 [45812.079559] [] ? __wake_up_common+0x82/0x120 [45812.086177] [] ? __wake_up_common_lock+0x91/0xc0 [45812.093201] [] core_sys_select+0x1ff/0x340 [45812.099631] [] ? __enqueue_entity+0x78/0x80 [45812.106160] [] ? put_prev_entity+0x8b/0x400 [45812.112687] [] ? pick_next_entity+0xa9/0x190 [45812.119325] [] ? set_next_entity+0x3c/0xe0 [45812.125771] [] ? __switch_to+0x151/0x580 [45812.132007] [] SyS_select+0xca/0x120 [45812.137864] [] ? system_call_after_swapgs+0x96/0x13a [45812.145280] [] ? system_call_after_swapgs+0xa2/0x13a [45812.152688] [] ? system_call_after_swapgs+0x96/0x13a [45812.160105] [] ? system_call_after_swapgs+0xa2/0x13a [45812.167507] [] system_call_fastpath+0x25/0x2a [45812.174242] [] ? system_call_after_swapgs+0xa2/0x13a [45812.181655] bash R running task 0 78778 78771 0x00000088 [45812.189605] Call Trace: [45812.192348] [] sched_show_task+0xa8/0x110 [45812.198689] [] show_state_filter+0x82/0xd0 [45812.205128] [] sysrq_handle_showstate+0x10/0x20 [45812.212049] [] __handle_sysrq+0x10d/0x170 [45812.218395] [] write_sysrq_trigger+0x28/0x40 [45812.225030] [] proc_reg_write+0x40/0x80 [45812.231168] [] vfs_write+0xc0/0x1f0 [45812.236926] [] ? system_call_after_swapgs+0xa2/0x13a [45812.244339] [] SyS_write+0x7f/0xf0 [45812.250009] [] ? system_call_after_swapgs+0xa2/0x13a [45812.257423] [] system_call_fastpath+0x25/0x2a [45812.264145] [] ? system_call_after_swapgs+0xa2/0x13a [45812.271567] Sched Debug Version: v0.11, 3.10.0-1160.49.1.el7_lustre.x86_64 #1 [45812.279560] ktime : 45810016.138521 [45812.286764] sched_clk : 45812271.564702 [45812.293974] cpu_clk : 45812271.564919 [45812.301178] jiffies : 4340477342 [45812.307903] sched_clock_stable() : 1 [45812.315454] sysctl_sched [45812.318300] .sysctl_sched_latency : 24.000000 [45812.325212] .sysctl_sched_min_granularity : 3.000000 [45812.332028] .sysctl_sched_wakeup_granularity : 4.000000 [45812.338842] .sysctl_sched_child_runs_first : 0 [45812.344986] .sysctl_sched_features : 56955 [45812.351516] .sysctl_sched_tunable_scaling : 1 (logaritmic) [45812.360588] cpu#0, 2693.251 MHz [45812.364113] .nr_running : 0 [45812.369295] .load : 0 [45812.374492] .nr_switches : 33471260 [45812.380351] .nr_load_updates : 6474043 [45812.386100] .nr_uninterruptible : -1262 [45812.391655] .next_balance : 4340.477429 [45812.397791] .curr->pid : 0 [45812.402974] .clock : 45812402.448215 [45812.409521] .cpu_load[0] : 0 [45812.414694] .cpu_load[1] : 0 [45812.419861] .cpu_load[2] : 0 [45812.425043] .cpu_load[3] : 0 [45812.430216] .cpu_load[4] : 0 [45812.435398] .avg_idle : 1000000 [45812.441146] .max_idle_balance_cost : 500000 [45812.448500] cfs_rq[0]:/ [45812.451236] .exec_clock : 0.000000 [45812.457090] .MIN_vruntime : 0.000001 [45812.462949] .min_vruntime : 280453.923107 [45812.469295] .max_vruntime : 0.000001 [45812.475147] .spread : 0.000000 [45812.480993] .spread0 : 0.000000 [45812.486839] .nr_spread_over : 0 [45812.492006] .nr_running : 0 [45812.497186] .load : 0 [45812.502352] .runnable_load_avg : 0 [45812.507531] .blocked_load_avg : 0 [45812.512719] .tg_load_avg : 0 [45812.517889] .tg_load_contrib : 0 [45812.523061] .tg_runnable_contrib : 0 [45812.528233] .tg->runnable_avg : 0 [45812.533413] .tg->cfs_bandwidth.timer_active: 0 [45812.538594] .throttled : 0 [45812.543764] .throttle_count : 0 [45812.548937] .avg->runnable_avg_sum : 8 [45812.554104] .avg->runnable_avg_period : 46485 [45812.561338] rt_rq[0]:/ [45812.563999] .rt_nr_running : 0 [45812.569179] .rt_throttled : 0 [45812.574366] .rt_time : 0.000000 [45812.580215] .rt_runtime : 950.000000 [45812.587957] runnable tasks: [45812.591093] task PID tree-key switches prio wait-time sum-exec sum-sleep [45812.603142] ---------------------------------------------------------------------------------------------------------- [45812.615095] kworker/0:0H 4 9211.787192 8 100 0.000000 0.105386 0.000000 0 / [45812.627540] ksoftirqd/0 6 280291.918030 71662 120 0.000000 1844.095009 0.000000 0 / [45812.640021] migration/0 7 0.000000 419 0 0.000000 1111.888154 0.000000 0 / [45812.652463] rcu_bh 8 101.146372 2 120 0.000000 0.003211 0.000000 0 / [45812.664930] lru-add-drain 10 104.649680 2 100 0.000000 0.004705 0.000000 0 / [45812.677372] watchdog/0 11 0.000000 11457 0 0.000000 209.624795 0.000000 0 / [45812.689841] kswapd0 192 1837.873764 4 120 0.000000 0.053103 0.000000 0 / [45812.702325] kworker/u305:8 505 280441.981765 39211 120 0.000000 8723.210147 0.000000 0 / [45812.714788] kworker/0:1H 579 280441.923318 109802 100 0.000000 1909.493586 0.000000 0 / [45812.727248] multipathd 725 0.000000 43 0 0.000000 15.615546 0.000000 0 / [45812.739690] multipathd 732 0.000000 84652 0 0.000000 9092.078510 0.000000 0 / [45812.752170] irqbalance 1101 280445.282075 5167 120 0.000000 38979.227763 0.000000 0 / [45812.764644] sssd_nss 1302 280441.989041 26967 120 0.000000 3126.468029 0.000000 0 / [45812.777115] automount 1924 10287.979866 138 120 0.000000 135.862735 0.000000 0 / [45812.789574] automount 2172 10282.655564 18 120 0.000000 15.696913 0.000000 0 / [45812.802032] ptlrpc_hr00_006 3458 270873.179155 268141 120 0.000000 8711.625582 0.000000 0 / [45812.814494] kiblnd_sd_00_03 3481 280442.064360 19506840 120 0.000000 392774.121348 0.000000 0 / [45812.826969] ptlrpcd_00_01 3492 280442.129702 477977 120 0.000000 18930.442758 0.000000 0 / [45812.839437] ptlrpcd_00_08 3499 280442.101810 442597 120 0.000000 18756.025480 0.000000 0 / [45812.851877] ptlrpcd_00_10 3501 280441.941565 443305 120 0.000000 18599.465798 0.000000 0 / [45812.864331] ll_mgs_0000 3552 272337.297801 40816 120 0.000000 2706.044535 0.000000 0 / [45812.876773] mdt00_000 3560 280442.028043 842775 120 0.000000 67471.453491 0.000000 0 / [45812.889254] mdt_seqm_0000 3576 278933.580493 30 120 0.000000 0.787945 0.000000 0 / [45812.901672] mdt_seqm_0001 3577 168565.624492 34 120 0.000000 2.000400 0.000000 0 / [45812.914148] osp-pre-1-0 3598 280441.954655 8882 120 0.000000 375.424133 0.000000 0 / [45812.926570] osp-pre-6-0 3616 280442.034089 12839 120 0.000000 637.531333 0.000000 0 / [45812.939045] osp-pre-8-0 3620 280441.984395 11839 120 0.000000 662.021766 0.000000 0 / [45812.951517] osp-syn-11-0 3625 270919.598855 212992 120 0.000000 5886.591782 0.000000 0 / [45812.964000] osp-pre-11-0 3626 280441.954463 11683 120 0.000000 573.845718 0.000000 0 / [45812.976480] osp-pre-13-0 3630 280441.979007 12050 120 0.000000 609.198925 0.000000 0 / [45812.988954] lquota_wb_soake 3639 280277.964823 762 120 0.000000 19.902541 0.000000 0 / [45813.001419] mdt_rdpg00_008 5764 270461.638529 286378 120 0.000000 19226.762274 0.000000 0 / [45813.013899] mdt_out00_006 12981 278828.547918 403424 120 0.000000 32780.538572 0.000000 0 / [45813.026383] ldlm_cb00_008 25772 270044.295168 52985 120 0.000000 2885.995675 0.000000 0 / [45813.038848] kworker/0:0 59042 271953.192296 4 120 0.000000 0.095969 0.000000 0 / [45813.051309] kworker/0:2 67752 280442.266231 4653 120 0.000000 349.674092 0.000000 0 / [45813.063760] kworker/u304:2 71184 280292.096670 16523 120 0.000000 724.274767 0.000000 0 / [45813.077906] cpu#1, 2693.251 MHz [45813.081439] .nr_running : 0 [45813.086621] .load : 0 [45813.091794] .nr_switches : 23406787 [45813.097647] .nr_load_updates : 6037110 [45813.103410] .nr_uninterruptible : -1175 [45813.108964] .next_balance : 4340.478148 [45813.115109] .curr->pid : 0 [45813.120283] .clock : 45813119.477096 [45813.126809] .cpu_load[0] : 0 [45813.131982] .cpu_load[1] : 0 [45813.137162] .cpu_load[2] : 0 [45813.142328] .cpu_load[3] : 0 [45813.147502] .cpu_load[4] : 0 [45813.152692] .avg_idle : 705690 [45813.158355] .max_idle_balance_cost : 500000 [45813.165681] cfs_rq[1]:/ [45813.168424] .exec_clock : 0.000000 [45813.174270] .MIN_vruntime : 0.000001 [45813.180124] .min_vruntime : 270544.541546 [45813.186461] .max_vruntime : 0.000001 [45813.192307] .spread : 0.000000 [45813.198161] .spread0 : -9910.501830 [45813.204403] .nr_spread_over : 0 [45813.209602] .nr_running : 0 [45813.214784] .load : 0 [45813.218589] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [45813.231422] .runnable_load_avg : 0 [45813.236601] .blocked_load_avg : 1 [45813.241781] .tg_load_avg : 0 [45813.246952] .tg_load_contrib : 0 [45813.252122] .tg_runnable_contrib : 0 [45813.257303] .tg->runnable_avg : 0 [45813.262476] .tg->cfs_bandwidth.timer_active: 0 [45813.267649] .throttled : 0 [45813.272828] .throttle_count : 0 [45813.278007] .avg->runnable_avg_sum : 85 [45813.283288] .avg->runnable_avg_period : 47637 [45813.290553] rt_rq[1]:/ [45813.293206] .rt_nr_running : 0 [45813.298380] .rt_throttled : 0 [45813.303553] .rt_time : 0.000000 [45813.309397] .rt_runtime : 950.000000 [45813.317126] runnable tasks: [45813.320254] task PID tree-key switches prio wait-time sum-exec sum-sleep [45813.332314] ---------------------------------------------------------------------------------------------------------- [45813.344283] watchdog/1 12 -2.931695 11457 0 0.000000 187.441492 0.000000 0 / [45813.356713] migration/1 13 0.000000 425 0 0.000000 1541.094953 0.000000 0 / [45813.369177] ksoftirqd/1 14 270152.318457 48770 120 0.000000 1237.247552 0.000000 0 / [45813.381587] kworker/1:0H 16 1360.053670 10 100 0.000000 0.045542 0.000000 0 / [45813.394089] scsi_eh_3 486 1172.458824 22 120 0.000000 0.258419 0.000000 0 / [45813.406503] scsi_eh_6 492 1172.765637 22 120 0.000000 0.494466 0.000000 0 / [45813.418955] jbd2/sda1-8 589 270532.728231 13318 120 0.000000 923.119565 0.000000 0 / [45813.431414] multipathd 730 0.000000 9159 0 0.000000 308.922867 0.000000 0 / [45813.443866] multipathd 733 0.000000 254 0 0.000000 43.607641 0.000000 0 / [45813.456303] kworker/1:1H 836 270532.542135 95721 100 0.000000 1728.558366 0.000000 0 / [45813.468767] sshd 1801 270055.855140 5711 120 0.000000 1950.680375 0.000000 0 / [45813.481258] ptlrpc_hr00_001 3453 265048.086679 268352 120 0.000000 8690.629780 0.000000 0 / [45813.493722] ptlrpc_hr00_007 3459 265046.664689 268162 120 0.000000 8762.830625 0.000000 0 / [45813.506193] kiblnd_sd_00_02 3480 270532.843779 19506230 120 0.000000 392950.427785 0.000000 0 / [45813.518636] ptlrpcd_00_01 3492 270533.009119 477981 120 0.000000 18930.731741 0.000000 0 / [45813.531089] ptlrpcd_00_04 3495 270532.736054 444003 120 0.000000 18583.415519 0.000000 0 / [45813.543559] ptlrpcd_00_13 3504 270532.717821 479967 120 0.000000 18990.472230 0.000000 0 / [45813.556020] ll_mgs_0001 3553 266380.815848 27822 120 0.000000 1791.033633 0.000000 0 / [45813.568479] osp-pre-5-0 3614 270532.568734 10089 120 0.000000 475.630585 0.000000 0 / [45813.580963] osp-pre-8-0 3620 270532.589816 11840 120 0.000000 662.070036 0.000000 0 / [45813.593446] dist_txn-0 3641 264739.725562 848751 120 0.000000 61738.308666 0.000000 0 / [45813.605911] mdt00_005 3775 265168.766819 937249 120 0.000000 71266.980319 0.000000 0 / [45813.618378] ldlm_cn00_006 3794 260476.349625 249733 120 0.000000 14595.179049 0.000000 0 / [45813.630837] ldlm_bl_03 4960 269826.988232 213771 120 0.000000 7473.761544 0.000000 0 / [45813.643292] mdt00_011 5288 267142.735919 1277568 120 0.000000 107023.165440 0.000000 0 / [45813.655767] mdt00_013 7036 270448.612563 688101 120 0.000000 55167.862451 0.000000 0 / [45813.668267] mdt_out00_008 12983 260359.982349 249027 120 0.000000 18042.958802 0.000000 0 / [45813.680725] ldlm_cb00_006 22101 264120.197685 52438 120 0.000000 3193.300926 0.000000 0 / [45813.693180] mdt_out00_014 23209 260624.508537 129011 120 0.000000 8943.238998 0.000000 0 / [45813.705645] kworker/u311:2 34625 269826.883040 1002 120 0.000000 71.342624 0.000000 0 / [45813.718101] mdt_io00_007 42598 264634.124063 1463 120 0.000000 124.170195 0.000000 0 / [45813.730567] kworker/1:0 44368 269825.538576 38752 120 0.000000 1287.643978 0.000000 0 / [45813.743040] kworker/1:1 59353 270532.848009 1221 120 0.000000 9.270312 0.000000 0 / [45813.755509] pickup 72921 270506.956266 67 120 0.000000 23.501558 0.000000 0 / [45813.769650] cpu#2, 2693.251 MHz [45813.773177] .nr_running : 0 [45813.778351] .load : 0 [45813.783526] .nr_switches : 21191471 [45813.789385] .nr_load_updates : 6041273 [45813.795141] .nr_uninterruptible : -954 [45813.800599] .next_balance : 4340.478840 [45813.806755] .curr->pid : 0 [45813.811927] .clock : 45813811.513669 [45813.818450] .cpu_load[0] : 0 [45813.823630] .cpu_load[1] : 0 [45813.828796] .cpu_load[2] : 0 [45813.833970] .cpu_load[3] : 0 [45813.839136] .cpu_load[4] : 0 [45813.844303] .avg_idle : 1000000 [45813.850052] .max_idle_balance_cost : 500000 [45813.857396] cfs_rq[2]:/ [45813.860134] .exec_clock : 0.000000 [45813.866033] .MIN_vruntime : 0.000001 [45813.871887] .min_vruntime : 272947.455123 [45813.878225] .max_vruntime : 0.000001 [45813.884080] .spread : 0.000000 [45813.889941] .spread0 : -7510.474564 [45813.896176] .nr_spread_over : 0 [45813.901350] .nr_running : 0 [45813.906525] .load : 0 [45813.911699] .runnable_load_avg : 0 [45813.916881] .blocked_load_avg : 0 [45813.922046] .tg_load_avg : 0 [45813.927222] .tg_load_contrib : 0 [45813.932402] .tg_runnable_contrib : 0 [45813.937585] .tg->runnable_avg : 0 [45813.942756] .tg->cfs_bandwidth.timer_active: 0 [45813.947930] .throttled : 0 [45813.953109] .throttle_count : 0 [45813.958283] .avg->runnable_avg_sum : 4 [45813.963456] .avg->runnable_avg_period : 47616 [45813.970704] rt_rq[2]:/ [45813.973370] .rt_nr_running : 0 [45813.978564] .rt_throttled : 0 [45813.983741] .rt_time : 0.000000 [45813.989594] .rt_runtime : 950.000000 [45813.997334] runnable tasks: [45814.000461] task PID tree-key switches prio wait-time sum-exec sum-sleep [45814.012521] ---------------------------------------------------------------------------------------------------------- [45814.024485] watchdog/2 18 -5.926473 11455 0 0.000000 192.145179 0.000000 0 / [45814.036928] migration/2 19 0.000000 466 0 0.000000 1115.486396 0.000000 0 / [45814.049412] ksoftirqd/2 20 272788.543111 46175 120 0.000000 1151.608750 0.000000 0 / [45814.061854] kworker/2:0H 22 2209.695595 10 100 0.000000 0.105054 0.000000 0 / [45814.074316] scsi_eh_7 494 2084.192183 22 120 0.000000 0.361877 0.000000 0 / [45814.086742] systemd-udevd 727 272843.448090 1625 120 0.000000 1557.359188 0.000000 0 / [45814.099234] kworker/2:1H 912 272935.455550 97365 100 0.000000 1804.664403 0.000000 0 / [45814.111676] kworker/2:3 1040 272936.883720 100183 120 0.000000 11116.509454 0.000000 0 / [45814.124148] auditd 1067 272866.108725 45177 116 0.000000 4416.849486 0.000000 0 / [45814.136598] ptlrpc_hr00_002 3454 268196.722643 268174 120 0.000000 8742.724909 0.000000 0 / [45814.149075] ptlrpc_hr00_004 3456 268196.726100 268157 120 0.000000 8688.319323 0.000000 0 / [45814.161506] ptlrpcd_00_02 3493 272935.516529 442690 120 0.000000 18752.029556 0.000000 0 / [45814.173993] ptlrpcd_00_09 3500 272935.646673 478339 120 0.000000 19014.413349 0.000000 0 / [45814.186450] ptlrpcd_00_12 3503 272935.513555 441831 120 0.000000 18716.821566 0.000000 0 / [45814.198881] ptlrpcd_00_14 3505 272935.599549 442522 120 0.000000 18828.589036 0.000000 0 / [45814.211321] ptlrpcd_00_15 3506 272935.939015 478852 120 0.000000 18770.446821 0.000000 0 / [45814.223794] osp_up2-0 3599 265990.687389 1074204 120 0.000000 68115.273148 0.000000 0 / [45814.236284] osp-pre-3-0 3610 272935.505927 11956 120 0.000000 582.305317 0.000000 0 / [45814.248732] osp-pre-9-0 3622 272935.488283 10129 120 0.000000 476.137127 0.000000 0 / [45814.261198] mdt00_003 3654 268297.065639 881088 120 0.000000 70044.120054 0.000000 0 / [45814.273661] mdt00_007 3780 272936.371989 890045 120 0.000000 77220.114631 0.000000 0 / [45814.286113] ldlm_cn00_008 3799 271564.407006 347910 120 0.000000 20806.977278 0.000000 0 / [45814.298555] mdt_rdpg00_005 3846 270870.443027 250333 120 0.000000 16986.215356 0.000000 0 / [45814.311074] mdt00_010 5287 270907.252515 992319 120 0.000000 79423.995132 0.000000 0 / [45814.323545] mdt_rdpg00_010 17957 267798.657389 198056 120 0.000000 13235.037377 0.000000 0 / [45814.336004] ldlm_bl_08 22098 272935.536156 91257 120 0.000000 2848.569578 0.000000 0 / [45814.348472] kworker/2:0 56139 218770.034894 2 120 0.000000 0.005323 0.000000 0 / [45814.362623] cpu#3, 2693.251 MHz [45814.366160] .nr_running : 5 [45814.371342] .load : 3072 [45814.376807] .nr_switches : 18566947 [45814.382671] .nr_load_updates : 6254388 [45814.388425] .nr_uninterruptible : -522 [45814.393880] .next_balance : 4340.479458 [45814.400015] .curr->pid : 78778 [45814.405591] .clock : 45753197.184397 [45814.412126] .cpu_load[0] : 3072 [45814.417615] .cpu_load[1] : 3072 [45814.423079] .cpu_load[2] : 3072 [45814.428553] .cpu_load[3] : 3072 [45814.434025] .cpu_load[4] : 3072 [45814.439490] .avg_idle : 1000000 [45814.445244] .max_idle_balance_cost : 500000 [45814.452557] cfs_rq[3]:/ [45814.455328] .exec_clock : 0.000000 [45814.461185] .MIN_vruntime : 264636.668974 [45814.467515] .min_vruntime : 264648.668974 [45814.473853] .max_vruntime : 264636.717048 [45814.480189] .spread : 0.048074 [45814.486025] .spread0 : -15811.737709 [45814.492370] .nr_spread_over : 0 [45814.497556] .nr_running : 3 [45814.502730] .load : 3072 [45814.508195] .runnable_load_avg : 193 [45814.513552] .blocked_load_avg : 0 [45814.518717] .tg_load_avg : 0 [45814.523891] .tg_load_contrib : 0 [45814.529069] .tg_runnable_contrib : 0 [45814.534243] .tg->runnable_avg : 0 [45814.539418] .tg->cfs_bandwidth.timer_active: 0 [45814.544610] .throttled : 0 [45814.549775] .throttle_count : 0 [45814.554949] .avg->runnable_avg_sum : 9222 [45814.560413] .avg->runnable_avg_period : 47250 [45814.567625] rt_rq[3]:/ [45814.570277] .rt_nr_running : 1 [45814.575441] .rt_throttled : 0 [45814.580618] .rt_time : 0.000000 [45814.586482] .rt_runtime : 950.000000 [45814.594214] runnable tasks: [45814.597341] task PID tree-key switches prio wait-time sum-exec sum-sleep [45814.609390] ---------------------------------------------------------------------------------------------------------- [45814.621368] watchdog/3 23 -5.931867 11440 0 0.000000 195.464426 0.000000 0 / [45814.633840] migration/3 24 0.000000 480 0 0.000000 1317.728606 0.000000 0 / [45814.646282] ksoftirqd/3 25 264636.717048 51261 120 0.000000 1436.939652 0.000000 0 / [45814.658736] kworker/3:0H 27 285.610415 8 100 0.000000 0.041237 0.000000 0 / [45814.671201] kworker/3:1H 532 264624.556198 98614 100 0.000000 1924.037580 0.000000 0 / [45814.683676] bioset 885 361.262515 2 100 0.000000 0.007303 0.000000 0 / [45814.696152] kworker/u310:0 3447 883.691442 4 120 0.000000 0.139196 0.000000 0 / [45814.708631] ptlrpc_hr00_000 3451 260498.198804 268097 120 0.000000 8803.788547 0.000000 0 / [45814.721101] mdt_fld_0000 3578 263416.853345 19 120 0.000000 1.054767 0.000000 0 / [45814.733556] osp-syn-4-0 3611 260549.079025 220581 120 0.000000 5887.052100 0.000000 0 / [45814.746020] osp-syn-7-0 3617 260556.768578 213482 120 0.000000 5915.157600 0.000000 0 / [45814.758514] ldlm_cn00_004 3790 260109.697781 239001 120 0.000000 15308.380383 0.000000 0 / [45814.770982] mdt_rdpg00_002 3796 260677.101437 263053 120 0.000000 17985.639717 0.000000 0 / [45814.783472] mdt_out00_004 3857 264289.794999 389363 120 0.000000 31723.999797 0.000000 0 / [45814.795933] mdt00_009 4645 252842.763549 1069209 120 0.000000 87926.772963 0.000000 0 / [45814.808402] kworker/3:1 23848 261870.906677 40721 120 0.000000 1198.294453 0.000000 0 / [45814.820860] kworker/3:2 68185 264636.668974 7537 120 0.000000 283.985986 0.000000 0 / [45814.833322] R bash 78778 264647.201243 138 120 0.000000 58.956139 0.000000 0 / [45814.847451] cpu#4, 2693.251 MHz [45814.850968] .nr_running : 0 [45814.856135] .load : 0 [45814.861313] .nr_switches : 17882435 [45814.867226] .nr_load_updates : 6036504 [45814.872990] .nr_uninterruptible : -408 [45814.878463] .next_balance : 4340.479918 [45814.884627] .curr->pid : 0 [45814.889801] .clock : 45814889.561935 [45814.896335] .cpu_load[0] : 0 [45814.901516] .cpu_load[1] : 32 [45814.906778] .cpu_load[2] : 13 [45814.912041] .cpu_load[3] : 14 [45814.917313] .cpu_load[4] : 13 [45814.922578] .avg_idle : 1000000 [45814.928334] .max_idle_balance_cost : 500000 [45814.935647] cfs_rq[4]:/ [45814.938397] .exec_clock : 0.000000 [45814.944256] .MIN_vruntime : 0.000001 [45814.950108] .min_vruntime : 253142.050640 [45814.956454] .max_vruntime : 0.000001 [45814.962307] .spread : 0.000000 [45814.968159] .spread0 : -27318.479635 [45814.974495] .nr_spread_over : 0 [45814.979672] .nr_running : 0 [45814.984845] .load : 0 [45814.990023] .runnable_load_avg : 0 [45814.995197] .blocked_load_avg : 26 [45815.000473] .tg_load_avg : 0 [45815.005644] .tg_load_contrib : 0 [45815.010817] .tg_runnable_contrib : 0 [45815.015987] .tg->runnable_avg : 0 [45815.021168] .tg->cfs_bandwidth.timer_active: 0 [45815.026340] .throttled : 0 [45815.031497] .throttle_count : 0 [45815.036686] .avg->runnable_avg_sum : 1426 [45815.042150] .avg->runnable_avg_period : 47821 [45815.049394] rt_rq[4]:/ [45815.052047] .rt_nr_running : 0 [45815.057222] .rt_throttled : 0 [45815.062402] .rt_time : 0.018970 [45815.068249] .rt_runtime : 950.000000 [45815.075963] runnable tasks: [45815.079094] task PID tree-key switches prio wait-time sum-exec sum-sleep [45815.091158] ---------------------------------------------------------------------------------------------------------- [45815.103117] watchdog/4 28 -8.925999 11456 0 0.000000 191.348377 0.000000 0 / [45815.115569] migration/4 29 0.000000 530 0 0.000000 1149.324306 0.000000 0 / [45815.128048] ksoftirqd/4 30 252247.761253 44769 120 0.000000 1259.170461 0.000000 0 / [45815.140503] kworker/4:0H 32 166.214574 7 100 0.000000 0.032499 0.000000 0 / [45815.152966] kworker/4:1H 533 252599.787890 95874 100 0.000000 1859.654066 0.000000 0 / [45815.165418] kworker/u308:0 539 252606.626859 5611 100 0.000000 191.843821 0.000000 0 / [45815.177888] kdmwork-253:0 865 251671.737192 18743 120 0.000000 556.276630 0.000000 0 / [45815.190357] kdmwork-253:1 872 251669.737667 92 120 0.000000 3.529945 0.000000 0 / [45815.202832] kdmflush 880 206.629269 2 100 0.000000 0.008232 0.000000 0 / [45815.215267] bioset 881 206.631565 2 100 0.000000 0.007070 0.000000 0 / [45815.227709] kdmflush 884 206.639543 2 100 0.000000 0.008029 0.000000 0 / [45815.240162] gssproxy 1175 252987.479528 1593 120 0.000000 76.343618 0.000000 0 / [45815.252639] rpc.gssd 1209 741.240246 55 120 0.000000 8.885296 0.000000 0 / [45815.265135] sssd_pac 1305 253017.347042 9160 120 0.000000 402.361986 0.000000 0 / [45815.277576] in:imjournal 1839 253148.684872 79508 120 0.000000 7723.871236 0.000000 0 / [45815.290038] automount 2009 253029.820949 2439 120 0.000000 133.224623 0.000000 0 / [45815.302523] kiblnd_connd 3477 253134.612640 47606 120 0.000000 3199.785054 0.000000 0 / [45815.314981] ptlrpcd_00_00 3491 253093.064014 444288 120 0.000000 18729.130270 0.000000 0 / [45815.327453] ptlrpcd_00_11 3502 253057.262437 479966 120 0.000000 18934.280218 0.000000 0 / [45815.339908] ldlm_cb00_001 3539 253037.373226 67497 120 0.000000 4178.902832 0.000000 0 / [45815.352363] osp-syn-10-0 3623 249472.919211 220749 120 0.000000 5867.380495 0.000000 0 / [45815.364849] osp-pre-13-0 3630 253114.821025 12051 120 0.000000 609.254092 0.000000 0 / [45815.377305] osp-pre-14-0 3632 253086.945215 13031 120 0.000000 695.968662 0.000000 0 / [45815.389748] osp-syn-15-0 3633 249472.120930 213196 120 0.000000 5886.880141 0.000000 0 / [45815.402194] mdt00_004 3767 253005.806886 558130 120 0.000000 49394.010050 0.000000 0 / [45815.414658] mdt_out00_003 3855 243778.382208 359815 120 0.000000 27097.064877 0.000000 0 / [45815.427150] ldlm_bl_06 5766 252609.942684 207127 120 0.000000 7418.013802 0.000000 0 / [45815.439592] ldlm_cb00_003 9903 248339.098311 63339 120 0.000000 4058.901848 0.000000 0 / [45815.452067] mdt_out00_010 13022 243677.117806 392816 120 0.000000 29619.591606 0.000000 0 / [45815.464529] mdt_out00_012 13062 253137.555360 418253 120 0.000000 34484.653771 0.000000 0 / [45815.476986] kworker/4:2 35102 253130.527010 44854 120 0.000000 1018.368643 0.000000 0 / [45815.489414] ldlm_cb00_009 35423 247725.045492 44510 120 0.000000 2535.014471 0.000000 0 / [45815.501852] kworker/4:1 56137 202183.188181 2 120 0.000000 0.005932 0.000000 0 / [45815.516012] cpu#5, 2693.251 MHz [45815.519537] .nr_running : 0 [45815.524730] .load : 0 [45815.529903] .nr_switches : 16637347 [45815.535750] .nr_load_updates : 6093661 [45815.541506] .nr_uninterruptible : 198 [45815.546879] .next_balance : 4340.480586 [45815.553022] .curr->pid : 0 [45815.558194] .clock : 45815557.588795 [45815.564724] .cpu_load[0] : 0 [45815.569881] .cpu_load[1] : 0 [45815.575047] .cpu_load[2] : 0 [45815.580233] .cpu_load[3] : 0 [45815.585414] .cpu_load[4] : 0 [45815.590571] .avg_idle : 1000000 [45815.596319] .max_idle_balance_cost : 500000 [45815.603668] cfs_rq[5]:/ [45815.606416] .exec_clock : 0.000000 [45815.612269] .MIN_vruntime : 0.000001 [45815.618123] .min_vruntime : 231113.819091 [45815.624459] .max_vruntime : 0.000001 [45815.630302] .spread : 0.000000 [45815.636138] .spread0 : -49346.711184 [45815.642490] .nr_spread_over : 0 [45815.647668] .nr_running : 0 [45815.652833] .load : 0 [45815.657999] .runnable_load_avg : 0 [45815.663179] .blocked_load_avg : 3 [45815.668351] .tg_load_avg : 0 [45815.673515] .tg_load_contrib : 0 [45815.678679] .tg_runnable_contrib : 0 [45815.683850] .tg->runnable_avg : 0 [45815.689029] .tg->cfs_bandwidth.timer_active: 0 [45815.694210] .throttled : 0 [45815.699383] .throttle_count : 0 [45815.704563] .avg->runnable_avg_sum : 211 [45815.709935] .avg->runnable_avg_period : 47038 [45815.717185] rt_rq[5]:/ [45815.719839] .rt_nr_running : 0 [45815.725014] .rt_throttled : 0 [45815.730188] .rt_time : 0.000000 [45815.736032] .rt_runtime : 950.000000 [45815.743747] runnable tasks: [45815.746882] task PID tree-key switches prio wait-time sum-exec sum-sleep [45815.758940] ---------------------------------------------------------------------------------------------------------- [45815.770893] rcu_sched 9 231112.400702 6236863 120 0.000000 165611.630642 0.000000 0 / [45815.783359] watchdog/5 33 -8.932000 11456 0 0.000000 228.235350 0.000000 0 / [45815.795813] migration/5 34 0.000000 408 0 0.000000 1101.336290 0.000000 0 / [45815.808280] ksoftirqd/5 35 231087.335831 47327 120 0.000000 1300.003367 0.000000 0 / [45815.820722] kworker/5:0H 37 124.376534 7 100 0.000000 0.037412 0.000000 0 / [45815.833206] kworker/5:1 233 231101.852641 94736 120 0.000000 2229.521121 0.000000 0 / [45815.845664] ata_sff 464 45.937391 2 100 0.000000 0.009224 0.000000 0 / [45815.858157] scsi_eh_4 488 117.236631 22 120 0.000000 0.369145 0.000000 0 / [45815.870665] kworker/5:1H 531 230521.038656 102782 100 0.000000 1930.660816 0.000000 0 / [45815.883151] ptlrpcd_rcv 3490 231076.858119 11817 120 0.000000 1552.939816 0.000000 0 / [45815.895620] ptlrpcd_00_03 3494 231085.592904 478631 120 0.000000 18820.673542 0.000000 0 / [45815.908128] ptlrpcd_00_05 3496 231090.873953 479110 120 0.000000 18949.943701 0.000000 0 / [45815.920580] ptlrpcd_00_06 3497 231101.873886 442743 120 0.000000 18733.732864 0.000000 0 / [45815.933038] mdt_out00_001 3571 230428.005771 409231 120 0.000000 32282.692387 0.000000 0 / [45815.945535] mdt_fld_0001 3579 229918.751232 108 120 0.000000 6.377479 0.000000 0 / [45815.958016] osp_up3-0 3601 227106.355471 670702 120 0.000000 43480.012660 0.000000 0 / [45815.970501] lquota_wb_soake 3640 230804.196704 762 120 0.000000 18.547703 0.000000 0 / [45815.982978] mdt_out00_002 3653 231091.694011 307159 120 0.000000 22587.398432 0.000000 0 / [45815.995446] ldlm_cn00_005 3792 231034.091896 313938 120 0.000000 19525.036751 0.000000 0 / [45816.007898] mdt_rdpg00_006 3853 227968.531581 285243 120 0.000000 18650.454858 0.000000 0 / [45816.020370] mdt_io00_003 4528 227353.902385 22633 120 0.000000 1882.879354 0.000000 0 / [45816.032822] mdt00_015 8312 231082.549755 914916 120 0.000000 73384.453873 0.000000 0 / [45816.045302] mdt_out00_009 12988 223548.859333 324470 120 0.000000 26173.575193 0.000000 0 / [45816.057746] ldlm_cn00_013 17013 223361.199220 205757 120 0.000000 11858.256928 0.000000 0 / [45816.070201] ldlm_cn00_015 19569 231018.594708 216677 120 0.000000 12434.880887 0.000000 0 / [45816.082665] mdt_out00_015 24983 223478.421483 266380 120 0.000000 20890.736051 0.000000 0 / [45816.095154] mdt_fld_0002 36795 229916.874426 123 120 0.000000 8.023224 0.000000 0 / [45816.107627] kworker/5:2 56099 181838.511432 2 120 0.000000 0.053085 0.000000 0 / [45816.121775] cpu#6, 2693.251 MHz [45816.125300] .nr_running : 1 [45816.130468] .load : 1024 [45816.135925] .nr_switches : 26622119 [45816.141776] .nr_load_updates : 6700589 [45816.147529] .nr_uninterruptible : 112 [45816.152889] .next_balance : 4340.481192 [45816.159038] .curr->pid : 0 [45816.164203] .clock : 45816164.199297 [45816.170745] .cpu_load[0] : 0 [45816.175917] .cpu_load[1] : 0 [45816.181089] .cpu_load[2] : 0 [45816.186251] .cpu_load[3] : 0 [45816.191422] .cpu_load[4] : 0 [45816.196592] .avg_idle : 1000000 [45816.202355] .max_idle_balance_cost : 500000 [45816.209687] cfs_rq[6]:/ [45816.212437] .exec_clock : 0.000000 [45816.218282] .MIN_vruntime : 239978.838362 [45816.224629] .min_vruntime : 239978.838362 [45816.230966] .max_vruntime : 239978.838362 [45816.237304] .spread : 0.000000 [45816.243157] .spread0 : -40481.691913 [45816.249503] .nr_spread_over : 0 [45816.254684] .nr_running : 1 [45816.259857] .load : 1024 [45816.265322] .runnable_load_avg : 31 [45816.270587] .blocked_load_avg : 0 [45816.275753] .tg_load_avg : 0 [45816.280932] .tg_load_contrib : 0 [45816.286105] .tg_runnable_contrib : 0 [45816.291276] .tg->runnable_avg : 0 [45816.296446] .tg->cfs_bandwidth.timer_active: 0 [45816.301619] .throttled : 0 [45816.306790] .throttle_count : 0 [45816.311951] .avg->runnable_avg_sum : 1500 [45816.317404] .avg->runnable_avg_period : 47672 [45816.324636] rt_rq[6]:/ [45816.327282] .rt_nr_running : 0 [45816.332457] .rt_throttled : 0 [45816.337635] .rt_time : 0.000000 [45816.343490] .rt_runtime : 950.000000 [45816.351238] runnable tasks: [45816.354381] task PID tree-key switches prio wait-time sum-exec sum-sleep [45816.366441] ---------------------------------------------------------------------------------------------------------- [45816.378406] watchdog/6 38 -8.925313 11456 0 0.000000 185.326112 0.000000 0 / [45816.390874] migration/6 39 0.000000 648 0 0.000000 1115.531582 0.000000 0 / [45816.403328] ksoftirqd/6 40 238472.202070 44752 120 0.000000 1027.235554 0.000000 0 / [45816.415786] kworker/6:0H 42 25.467606 7 100 0.000000 0.039730 0.000000 0 / [45816.428277] kdevtmpfs 171 742.661019 257 120 0.000000 5.067976 0.000000 0 / [45816.440733] scsi_eh_2 484 24.198010 22 120 0.000000 0.531074 0.000000 0 / [45816.453196] kworker/6:1H 529 238456.453936 103148 100 0.000000 1833.738847 0.000000 0 / [45816.465651] systemd-journal 672 239985.136739 32160 120 0.000000 15346.452518 0.000000 0 / [45816.478175] sssd_be 1257 239962.456128 11062 120 0.000000 1544.757300 0.000000 0 / [45816.490647] automount 2010 239818.484748 7317 120 0.000000 363.738023 0.000000 0 / [45816.503107] kworker/6:2 3528 239960.776819 94241 120 0.000000 2699.843326 0.000000 0 / [45816.515569] ldlm_bl_02 3549 238496.757941 218507 120 0.000000 7663.428933 0.000000 0 / [45816.528028] mdt_rdpg00_001 3567 234513.630665 216070 120 0.000000 14205.052065 0.000000 0 / [45816.540503] mdt_out00_000 3570 238240.922822 398153 120 0.000000 32688.113321 0.000000 0 / [45816.552974] osp-pre-3-0 3602 239960.966688 8922 120 0.000000 372.078648 0.000000 0 / [45816.565427] mdt_rdpg00_003 3801 234940.648743 233696 120 0.000000 16203.284050 0.000000 0 / [45816.577893] ldlm_bl_09 22099 238496.662688 91258 120 0.000000 2858.119415 0.000000 0 / [45816.590373] mdt00_017 22104 234894.709947 171262 120 0.000000 16335.106603 0.000000 0 / [45816.602821] ldlm_cb00_007 22183 234032.982956 65856 120 0.000000 3889.453054 0.000000 0 / [45816.615280] mdt_io00_006 30183 234449.511923 4093 120 0.000000 211.299002 0.000000 0 / [45816.627729] ldlm_bl_11 35424 238496.675715 51784 120 0.000000 1449.441247 0.000000 0 / [45816.640187] ldlm_bl_16 36869 238496.819057 48547 120 0.000000 1337.297555 0.000000 0 / [45816.652642] mdt_fld_0003 44997 237355.946267 31 120 0.000000 1.800472 0.000000 0 / [45816.665113] kworker/u311:1 60143 239895.627919 1048 120 0.000000 108.981732 0.000000 0 / [45816.677585] kworker/6:0 64532 232178.150545 2 120 0.000000 0.009554 0.000000 0 / [45816.691720] cpu#7, 2693.251 MHz [45816.695237] .nr_running : 0 [45816.700403] .load : 0 [45816.705569] .nr_switches : 30102729 [45816.711409] .nr_load_updates : 6737414 [45816.717157] .nr_uninterruptible : -391 [45816.722628] .next_balance : 4340.481762 [45816.728776] .curr->pid : 0 [45816.733956] .clock : 45816733.645903 [45816.740475] .cpu_load[0] : 0 [45816.745636] .cpu_load[1] : 0 [45816.750804] .cpu_load[2] : 0 [45816.755982] .cpu_load[3] : 0 [45816.761162] .cpu_load[4] : 0 [45816.766341] .avg_idle : 1000000 [45816.772100] .max_idle_balance_cost : 500000 [45816.779452] cfs_rq[7]:/ [45816.782200] .exec_clock : 0.000000 [45816.788053] .MIN_vruntime : 0.000001 [45816.793914] .min_vruntime : 216149.506078 [45816.800252] .max_vruntime : 0.000001 [45816.806105] .spread : 0.000000 [45816.811949] .spread0 : -64311.024197 [45816.818288] .nr_spread_over : 0 [45816.823458] .nr_running : 0 [45816.828637] .load : 0 [45816.833815] .runnable_load_avg : 0 [45816.838988] .blocked_load_avg : 20 [45816.844259] .tg_load_avg : 0 [45816.849432] .tg_load_contrib : 0 [45816.854607] .tg_runnable_contrib : 0 [45816.859788] .tg->runnable_avg : 0 [45816.864970] .tg->cfs_bandwidth.timer_active: 0 [45816.870157] .throttled : 0 [45816.875333] .throttle_count : 0 [45816.880491] .avg->runnable_avg_sum : 1074 [45816.885956] .avg->runnable_avg_period : 46409 [45816.893201] rt_rq[7]:/ [45816.895845] .rt_nr_running : 0 [45816.901020] .rt_throttled : 0 [45816.906193] .rt_time : 0.000000 [45816.912038] .rt_runtime : 950.000000 [45816.919758] runnable tasks: [45816.922895] task PID tree-key switches prio wait-time sum-exec sum-sleep [45816.934946] ---------------------------------------------------------------------------------------------------------- [45816.946908] watchdog/7 43 -8.924982 11456 0 0.000000 199.177084 0.000000 0 / [45816.959358] migration/7 44 0.000000 452 0 0.000000 1120.183261 0.000000 0 / [45816.971802] ksoftirqd/7 45 215659.000755 46952 120 0.000000 1054.346110 0.000000 0 / [45816.984260] kworker/7:0H 47 59.370679 7 100 0.000000 0.043804 0.000000 0 / [45816.996719] kworker/7:1 234 216140.992278 97315 120 0.000000 2336.922502 0.000000 0 / [45817.009187] kauditd 340 215759.631332 49421 120 0.000000 1897.715770 0.000000 0 / [45817.021661] kworker/7:1H 530 215999.198605 103904 100 0.000000 1696.187091 0.000000 0 / [45817.034127] kdmwork-253:4 886 110.625360 35 120 0.000000 0.245062 0.000000 0 / [45817.046608] gmain 1238 216114.651501 11452 120 0.000000 687.332327 0.000000 0 / [45817.059080] rs:main Q:Reg 1840 216153.977203 32859 120 0.000000 3829.335046 0.000000 0 / [45817.071526] ptlrpc_hr00_003 3455 212046.409250 268132 120 0.000000 8776.031424 0.000000 0 / [45817.083987] ldlm_bl_01 3548 215680.678650 218576 120 0.000000 7702.507200 0.000000 0 / [45817.096433] mdt_io00_001 3581 211656.993341 21478 120 0.000000 1961.725204 0.000000 0 / [45817.108872] osp-pre-15-0 3634 216109.984360 11622 120 0.000000 554.432205 0.000000 0 / [45817.121352] ldlm_cb00_002 9901 216093.389800 78521 120 0.000000 4897.199302 0.000000 0 / [45817.133801] ldlm_cn00_012 16973 209935.897945 253356 120 0.000000 13813.170552 0.000000 0 / [45817.146229] ldlm_cn00_016 23207 216093.587819 161869 120 0.000000 9563.275553 0.000000 0 / [45817.158684] ldlm_cn00_019 24514 216093.549707 139478 120 0.000000 7477.000536 0.000000 0 / [45817.171147] kworker/7:2 29181 102861.754482 2 120 0.000000 0.067287 0.000000 0 / [45817.183606] ldlm_cn00_020 33154 211703.987497 123457 120 0.000000 6986.328953 0.000000 0 / [45817.196059] ldlm_bl_13 36866 215680.679686 48535 120 0.000000 1342.103442 0.000000 0 / [45817.208509] kworker/u304:0 71789 216140.973430 14320 120 0.000000 754.392521 0.000000 0 / [45817.222621] cpu#8, 2693.251 MHz [45817.226147] .nr_running : 0 [45817.231312] .load : 0 [45817.236487] .nr_switches : 37827062 [45817.242345] .nr_load_updates : 9356207 [45817.248096] .nr_uninterruptible : -2026 [45817.253659] .next_balance : 4340.482293 [45817.259797] .curr->pid : 0 [45817.264970] .clock : 45817264.680433 [45817.271493] .cpu_load[0] : 0 [45817.276668] .cpu_load[1] : 0 [45817.281840] .cpu_load[2] : 0 [45817.286997] .cpu_load[3] : 0 [45817.292171] .cpu_load[4] : 0 [45817.297338] .avg_idle : 1000000 [45817.303105] .max_idle_balance_cost : 500000 [45817.310449] cfs_rq[8]:/ [45817.313204] .exec_clock : 0.000000 [45817.319056] .MIN_vruntime : 0.000001 [45817.324902] .min_vruntime : 451060.706482 [45817.331239] .max_vruntime : 0.000001 [45817.337092] .spread : 0.000000 [45817.342941] .spread0 : 170600.176207 [45817.349287] .nr_spread_over : 0 [45817.354461] .nr_running : 0 [45817.359629] .load : 0 [45817.364806] .runnable_load_avg : 0 [45817.369986] .blocked_load_avg : 0 [45817.375153] .tg_load_avg : 0 [45817.380329] .tg_load_contrib : 0 [45817.385496] .tg_runnable_contrib : 0 [45817.390670] .tg->runnable_avg : 0 [45817.395834] .tg->cfs_bandwidth.timer_active: 0 [45817.401006] .throttled : 0 [45817.406184] .throttle_count : 0 [45817.411363] .avg->runnable_avg_sum : 0 [45817.416535] .avg->runnable_avg_period : 47747 [45817.423781] rt_rq[8]:/ [45817.426423] .rt_nr_running : 0 [45817.431595] .rt_throttled : 0 [45817.436768] .rt_time : 0.000000 [45817.442640] .rt_runtime : 950.000000 [45817.450364] runnable tasks: [45817.453501] task PID tree-key switches prio wait-time sum-exec sum-sleep [45817.465544] ---------------------------------------------------------------------------------------------------------- [45817.477506] systemd 1 451060.706482 59865 120 0.000000 28865.877662 0.000000 1 / [45817.489955] watchdog/8 48 -11.906928 11456 0 0.000000 224.276829 0.000000 1 / [45817.502426] migration/8 49 0.000000 386 0 0.000000 3239.554946 0.000000 1 / [45817.514858] ksoftirqd/8 50 451048.736374 110282 120 0.000000 2414.660998 0.000000 1 / [45817.527302] kworker/8:0H 52 474.593969 10 100 0.000000 0.057423 0.000000 1 / [45817.539778] kworker/8:1H 948 451048.706680 182680 100 0.000000 3704.035159 0.000000 1 / [45817.552240] rpcbind 1096 451048.813570 1540 120 0.000000 130.976034 0.000000 1 / [45817.564726] master 1925 451048.929590 2387 120 0.000000 238.307615 0.000000 1 / [45817.577208] agetty 1994 1010.901959 16 120 0.000000 9.099728 0.000000 1 / [45817.589681] ptlrpc_hr01_001 3461 445978.755136 442950 120 0.000000 17824.638370 0.000000 1 / [45817.602145] ptlrpcd_01_11 3518 451049.183047 513923 120 0.000000 17617.557977 0.000000 1 / [45817.614612] mdt01_001 3564 438145.640993 1110715 120 0.000000 104784.954511 0.000000 1 / [45817.627081] mdt_io01_001 3584 443551.241973 39442 120 0.000000 3428.134030 0.000000 1 / [45817.639562] mdt_io01_002 3585 443567.153346 34458 120 0.000000 2994.349712 0.000000 1 / [45817.652050] mdt_out01_002 3768 451051.038973 797523 120 0.000000 55073.141384 0.000000 1 / [45817.664505] mdt01_006 3774 444386.734230 1804086 120 0.000000 170558.751966 0.000000 1 / [45817.676954] mdt01_013 5286 451048.795276 1863870 120 0.000000 177262.501192 0.000000 1 / [45817.689439] ldlm_cn01_016 7589 448135.764480 450570 120 0.000000 28560.430495 0.000000 1 / [45817.701878] mdt_out01_010 7629 440603.885702 395210 120 0.000000 24979.806938 0.000000 1 / [45817.714346] ldlm_cb01_005 9904 442208.907821 363496 120 0.000000 21349.161984 0.000000 1 / [45817.726803] ldlm_cn01_026 34149 451048.847564 156875 120 0.000000 9027.984852 0.000000 1 / [45817.739256] mdt01_026 35260 451049.691423 205657 120 0.000000 18742.154104 0.000000 1 / [45817.751742] kworker/8:1 62687 436883.548192 4 120 0.000000 0.137931 0.000000 1 / [45817.764242] kworker/8:0 64491 451048.781333 5740 120 0.000000 129.750037 0.000000 1 / [45817.778381] cpu#9, 2693.251 MHz [45817.781907] .nr_running : 0 [45817.787082] .load : 0 [45817.792249] .nr_switches : 37752519 [45817.798101] .nr_load_updates : 9296645 [45817.803849] .nr_uninterruptible : -1532 [45817.809410] .next_balance : 4340.482849 [45817.815553] .curr->pid : 0 [45817.820741] .clock : 45817820.700624 [45817.827272] .cpu_load[0] : 0 [45817.832443] .cpu_load[1] : 0 [45817.837617] .cpu_load[2] : 0 [45817.842783] .cpu_load[3] : 0 [45817.847962] .cpu_load[4] : 0 [45817.853134] .avg_idle : 1000000 [45817.858888] .max_idle_balance_cost : 500000 [45817.866237] cfs_rq[9]:/ [45817.868999] .exec_clock : 0.000000 [45817.874846] .MIN_vruntime : 0.000001 [45817.880708] .min_vruntime : 455875.687134 [45817.887054] .max_vruntime : 0.000001 [45817.892906] .spread : 0.000000 [45817.898759] .spread0 : 175415.156859 [45817.905095] .nr_spread_over : 0 [45817.910254] .nr_running : 0 [45817.915421] .load : 0 [45817.920581] .runnable_load_avg : 0 [45817.925766] .blocked_load_avg : 0 [45817.930934] .tg_load_avg : 0 [45817.936100] .tg_load_contrib : 0 [45817.941274] .tg_runnable_contrib : 0 [45817.946449] .tg->runnable_avg : 0 [45817.951622] .tg->cfs_bandwidth.timer_active: 0 [45817.956816] .throttled : 0 [45817.961989] .throttle_count : 0 [45817.967158] .avg->runnable_avg_sum : 0 [45817.972331] .avg->runnable_avg_period : 46738 [45817.979586] rt_rq[9]:/ [45817.982237] .rt_nr_running : 0 [45817.987402] .rt_throttled : 0 [45817.992567] .rt_time : 0.000000 [45817.998419] .rt_runtime : 950.000000 [45818.006148] runnable tasks: [45818.009290] task PID tree-key switches prio wait-time sum-exec sum-sleep [45818.021354] ---------------------------------------------------------------------------------------------------------- [45818.033336] watchdog/9 54 -11.903696 11456 0 0.000000 203.014991 0.000000 1 / [45818.045773] migration/9 55 0.000000 391 0 0.000000 1160.599096 0.000000 1 / [45818.058249] ksoftirqd/9 56 455858.153736 110097 120 0.000000 2454.069952 0.000000 1 / [45818.070703] kworker/9:0H 58 83.032371 9 100 0.000000 0.045015 0.000000 1 / [45818.083187] khugepaged 195 455875.687134 4584 139 0.000000 221.283093 0.000000 1 / [45818.095665] kworker/9:2 572 455854.949160 162624 120 0.000000 4210.520599 0.000000 1 / [45818.108110] kworker/9:1H 893 455860.890045 180280 100 0.000000 3693.888511 0.000000 1 / [45818.120577] polkitd 1291 455856.551427 28357 120 0.000000 3917.021173 0.000000 1 / [45818.133036] sssd_pam 1303 455863.723365 9164 120 0.000000 449.785421 0.000000 1 / [45818.145498] ptlrpc_hr01_003 3463 450835.048312 442907 120 0.000000 17808.370436 0.000000 1 / [45818.157971] ptlrpc_hr01_004 3464 450835.037917 442831 120 0.000000 17876.843736 0.000000 1 / [45818.170417] ptlrpcd_01_00 3507 455861.187578 472964 120 0.000000 17525.979462 0.000000 1 / [45818.182850] ptlrpcd_01_14 3521 455861.030909 473884 120 0.000000 17559.262562 0.000000 1 / [45818.195305] mdt01_002 3565 450980.236719 943732 120 0.000000 84406.374284 0.000000 1 / [45818.207771] mdt_rdpg01_001 3569 451974.515336 241387 120 0.000000 17948.801506 0.000000 1 / [45818.220228] mdt_out01_001 3573 455548.694936 648178 120 0.000000 44571.640542 0.000000 1 / [45818.232705] osp-pre-12-0 3628 455662.829581 12154 120 0.000000 643.604205 0.000000 1 / [45818.245165] mdt_out01_003 3772 455863.778058 1221030 120 0.000000 84403.789514 0.000000 1 / [45818.257625] mdt01_011 3787 455863.981493 1516298 120 0.000000 143049.764610 0.000000 1 / [45818.270093] mdt_rdpg01_008 3851 451928.244804 257019 120 0.000000 18984.997503 0.000000 1 / [45818.282552] mdt_out01_007 4874 455559.909606 985297 120 0.000000 66951.445660 0.000000 1 / [45818.295030] mdt01_014 5289 455865.108401 850445 120 0.000000 80512.602430 0.000000 1 / [45818.307483] ldlm_cn01_012 5445 455860.939550 437026 120 0.000000 26568.430202 0.000000 1 / [45818.319917] mdt01_016 7037 455864.566754 1256796 120 0.000000 118255.607224 0.000000 1 / [45818.332409] ldlm_cn01_024 34147 455836.734799 204213 120 0.000000 12160.211141 0.000000 1 / [45818.344877] mdt01_024 35217 453886.582919 337781 120 0.000000 30665.449698 0.000000 1 / [45818.357353] kworker/9:0 62761 431930.141515 2 120 0.000000 0.068241 0.000000 1 / [45818.371507] cpu#10, 2693.251 MHz [45818.375130] .nr_running : 0 [45818.380291] .load : 0 [45818.385458] .nr_switches : 37768946 [45818.391311] .nr_load_updates : 9463992 [45818.397060] .nr_uninterruptible : -2481 [45818.402628] .next_balance : 4340.483442 [45818.408778] .curr->pid : 0 [45818.413953] .clock : 45818413.732732 [45818.420478] .cpu_load[0] : 0 [45818.425646] .cpu_load[1] : 0 [45818.430833] .cpu_load[2] : 0 [45818.436007] .cpu_load[3] : 0 [45818.441180] .cpu_load[4] : 0 [45818.446347] .avg_idle : 922563 [45818.452007] .max_idle_balance_cost : 500000 [45818.459353] cfs_rq[10]:/ [45818.462204] .exec_clock : 0.000000 [45818.468057] .MIN_vruntime : 0.000001 [45818.473910] .min_vruntime : 457430.727722 [45818.480233] .max_vruntime : 0.000001 [45818.486077] .spread : 0.000000 [45818.491929] .spread0 : 176969.428168 [45818.498261] .nr_spread_over : 0 [45818.503478] .nr_running : 0 [45818.508651] .load : 0 [45818.513828] .runnable_load_avg : 0 [45818.519009] .blocked_load_avg : 0 [45818.524192] .tg_load_avg : 0 [45818.529373] .tg_load_contrib : 0 [45818.534539] .tg_runnable_contrib : 0 [45818.539714] .tg->runnable_avg : 0 [45818.544879] .tg->cfs_bandwidth.timer_active: 0 [45818.550052] .throttled : 0 [45818.555215] .throttle_count : 0 [45818.560384] .avg->runnable_avg_sum : 0 [45818.565550] .avg->runnable_avg_period : 46148 [45818.572796] rt_rq[10]:/ [45818.575521] .rt_nr_running : 0 [45818.580694] .rt_throttled : 0 [45818.585865] .rt_time : 0.000000 [45818.591712] .rt_runtime : 950.000000 [45818.599454] runnable tasks: [45818.602592] task PID tree-key switches prio wait-time sum-exec sum-sleep [45818.614641] ---------------------------------------------------------------------------------------------------------- [45818.626614] watchdog/10 59 -11.908619 11456 0 0.000000 214.638618 0.000000 1 / [45818.639078] migration/10 60 0.000000 470 0 0.000000 1126.914588 0.000000 1 / [45818.651534] ksoftirqd/10 61 457418.758292 107090 120 0.000000 2546.251974 0.000000 1 / [45818.663989] kworker/10:0H 63 192.145135 9 100 0.000000 0.083700 0.000000 1 / [45818.676437] kswapd1 193 -13.026341 3 120 0.000000 0.040290 0.000000 1 / [45818.688892] kworker/10:1 215 457418.794398 165697 120 0.000000 3221.780460 0.000000 1 / [45818.701356] kworker/u307:0 538 82.286496 3 100 0.000000 0.017598 0.000000 1 / [45818.713786] kworker/10:1H 894 457418.727781 174383 100 0.000000 3609.698049 0.000000 1 / [45818.726259] NetworkManager 1100 457430.727722 31771 120 0.000000 3589.427494 0.000000 1 / [45818.738733] ptlrpc_hr01_007 3467 452311.215197 442932 120 0.000000 17745.735051 0.000000 1 / [45818.751203] monitor_thread 3489 457420.137491 45701 120 0.000000 2205.873468 0.000000 1 / [45818.763668] ptlrpcd_01_03 3510 457419.240466 514425 120 0.000000 17518.574978 0.000000 1 / [45818.776110] ptlrpcd_01_04 3511 457419.356129 473708 120 0.000000 17423.071048 0.000000 1 / [45818.788568] ptlrpcd_01_12 3519 457419.523673 472904 120 0.000000 17541.838490 0.000000 1 / [45818.801009] ldlm_cb01_001 3541 448524.691091 308046 120 0.000000 17769.920806 0.000000 1 / [45818.813489] mdt_io01_000 3583 450621.573244 28818 120 0.000000 2600.683706 0.000000 1 / [45818.825918] osp-pre-0-0 3604 457419.172812 11036 120 0.000000 555.170945 0.000000 1 / [45818.838375] osp-pre-7-0 3618 457418.785487 10325 120 0.000000 462.837053 0.000000 1 / [45818.850807] mdt_rdpg01_002 3771 452780.196117 341155 120 0.000000 26489.841225 0.000000 1 / [45818.863305] mdt01_008 3779 454078.444258 1679171 120 0.000000 157421.301526 0.000000 1 / [45818.875735] mdt_rdpg01_003 3844 450262.028247 299432 120 0.000000 22403.007228 0.000000 1 / [45818.888203] mdt_rdpg01_009 3852 450265.849035 330320 120 0.000000 24642.503767 0.000000 1 / [45818.900665] ldlm_cb01_002 4961 456311.261514 315605 120 0.000000 18422.621742 0.000000 1 / [45818.913142] ldlm_cn01_014 7587 448725.120494 315205 120 0.000000 18458.290963 0.000000 1 / [45818.925602] mdt_out01_012 7670 453245.626996 720783 120 0.000000 47751.222254 0.000000 1 / [45818.938060] ldlm_cb01_007 22095 448511.268994 186540 120 0.000000 9899.637596 0.000000 1 / [45818.950521] ldlm_cb01_008 22096 448516.490554 211237 120 0.000000 11372.208892 0.000000 1 / [45818.962963] ldlm_cn01_021 34143 457419.038269 169007 120 0.000000 8866.548310 0.000000 1 / [45818.975437] ldlm_bl_15 36868 457414.189947 48539 120 0.000000 1353.815158 0.000000 1 / [45818.987887] kworker/10:2 63910 439905.758901 2 120 0.000000 0.007956 0.000000 1 / [45819.000349] kworker/u306:2 64293 456037.775677 6 120 0.000000 0.481710 0.000000 1 / [45819.012810] kworker/u312:1 68506 457301.120610 11 120 0.000000 0.877493 0.000000 1 / [45819.026998] cpu#11, 2693.251 MHz [45819.030626] .nr_running : 0 [45819.035825] .load : 0 [45819.040996] .nr_switches : 37571952 [45819.046867] .nr_load_updates : 9399294 [45819.052622] .nr_uninterruptible : -1553 [45819.058184] .next_balance : 4340.484097 [45819.064328] .curr->pid : 0 [45819.069502] .clock : 45819068.755295 [45819.076037] .cpu_load[0] : 0 [45819.081212] .cpu_load[1] : 0 [45819.086384] .cpu_load[2] : 0 [45819.091541] .cpu_load[3] : 0 [45819.096722] .cpu_load[4] : 0 [45819.101893] .avg_idle : 1000000 [45819.107640] .max_idle_balance_cost : 500000 [45819.114973] cfs_rq[11]:/ [45819.117822] .exec_clock : 0.000000 [45819.123674] .MIN_vruntime : 0.000001 [45819.129528] .min_vruntime : 448952.243093 [45819.135870] .max_vruntime : 0.000001 [45819.141716] .spread : 0.000000 [45819.147569] .spread0 : 168488.179764 [45819.153905] .nr_spread_over : 0 [45819.159091] .nr_running : 0 [45819.164264] .load : 0 [45819.169444] .runnable_load_avg : 0 [45819.174610] .blocked_load_avg : 0 [45819.179799] .tg_load_avg : 0 [45819.184964] .tg_load_contrib : 0 [45819.190125] .tg_runnable_contrib : 0 [45819.195285] .tg->runnable_avg : 0 [45819.200452] .tg->cfs_bandwidth.timer_active: 0 [45819.205626] .throttled : 0 [45819.210818] .throttle_count : 0 [45819.215985] .avg->runnable_avg_sum : 32 [45819.221255] .avg->runnable_avg_period : 47893 [45819.228515] rt_rq[11]:/ [45819.231271] .rt_nr_running : 0 [45819.236453] .rt_throttled : 0 [45819.241618] .rt_time : 0.023128 [45819.247465] .rt_runtime : 950.000000 [45819.255171] runnable tasks: [45819.258300] task PID tree-key switches prio wait-time sum-exec sum-sleep [45819.270349] ---------------------------------------------------------------------------------------------------------- [45819.282320] watchdog/11 64 -11.907589 11457 0 0.000000 208.166402 0.000000 1 / [45819.294770] migration/11 65 0.000000 420 0 0.000000 1321.845631 0.000000 1 / [45819.307237] ksoftirqd/11 66 448940.273452 108725 120 0.000000 2448.641834 0.000000 1 / [45819.319697] kworker/11:0H 68 283.981569 9 100 0.000000 0.052138 0.000000 1 / [45819.332172] kworker/11:1 254 448940.345960 162472 120 0.000000 3191.492955 0.000000 1 / [45819.344633] kipmi0 806 271.404195 457 139 0.000000 3.564646 0.000000 1 / [45819.357088] kworker/11:1H 813 448940.243570 176675 100 0.000000 3650.848188 0.000000 1 / [45819.369530] dbus-daemon 1093 448952.243093 83988 120 0.000000 20464.253307 0.000000 1 / [45819.381971] gssproxy 1191 679.591778 35 120 0.000000 10.320245 0.000000 1 / [45819.394432] sssd_ssh 1304 448940.824538 20560 120 0.000000 1397.529035 0.000000 1 / [45819.406891] munged 1829 448933.709448 806 120 0.000000 363.370543 0.000000 1 / [45819.419360] ptlrpc_hr01_006 3466 444112.007882 442878 120 0.000000 17732.588964 0.000000 1 / [45819.431819] kiblnd_sd_01_00 3482 448949.735724 44830771 120 0.000000 736689.762127 0.000000 1 / [45819.444295] ptlrpcd_01_02 3509 448940.575970 473974 120 0.000000 17425.918277 0.000000 1 / [45819.456742] ptlrpcd_01_09 3516 448941.130396 514809 120 0.000000 17567.251265 0.000000 1 / [45819.469229] osp-syn-0-0 3603 444133.767233 216272 120 0.000000 5858.861598 0.000000 1 / [45819.481710] osp-pre-4-0 3612 448940.290759 12075 120 0.000000 620.928722 0.000000 1 / [45819.494184] osp-syn-14-0 3631 444135.041853 221470 120 0.000000 5775.302038 0.000000 1 / [45819.506703] mdt01_009 3784 379938.966963 714721 120 0.000000 66176.488777 0.000000 1 / [45819.519159] mdt01_010 3785 444291.238151 1176310 120 0.000000 109080.526566 0.000000 1 / [45819.531613] mdt_rdpg01_005 3848 441979.579332 358795 120 0.000000 27441.262918 0.000000 1 / [45819.544090] mdt01_012 4646 287065.880780 1377150 120 0.000000 131746.090333 0.000000 1 / [45819.556573] ldlm_bl_04 5123 448916.926398 212231 120 0.000000 7493.555938 0.000000 1 / [45819.569029] mdt_out01_009 6643 448588.048287 776369 120 0.000000 50786.867758 0.000000 1 / [45819.581472] mdt_out01_011 7630 445199.972684 741529 120 0.000000 52625.280455 0.000000 1 / [45819.593925] ldlm_cb01_006 10177 440565.589987 343235 120 0.000000 20049.635053 0.000000 1 / [45819.606410] ldlm_cn01_023 34145 448916.519996 116642 120 0.000000 6619.425745 0.000000 1 / [45819.618883] mdt_out01_018 34307 448588.753174 483544 120 0.000000 32456.946897 0.000000 1 / [45819.631343] kworker/11:0 55268 371265.749891 4 120 0.000000 0.016041 0.000000 1 / [45819.643813] mdt_io01_011 64774 441831.149412 838 120 0.000000 49.737618 0.000000 1 / [45819.656253] kworker/u312:2 78186 448863.168989 4 120 0.000000 0.144223 0.000000 1 / [45819.670386] cpu#12, 2693.251 MHz [45819.674015] .nr_running : 0 [45819.679197] .load : 0 [45819.684379] .nr_switches : 37899695 [45819.690224] .nr_load_updates : 9338488 [45819.695971] .nr_uninterruptible : -1480 [45819.701525] .next_balance : 4340.484741 [45819.707684] .curr->pid : 0 [45819.712849] .clock : 45819712.792570 [45819.719380] .cpu_load[0] : 0 [45819.724555] .cpu_load[1] : 0 [45819.729729] .cpu_load[2] : 0 [45819.734917] .cpu_load[3] : 0 [45819.740076] .cpu_load[4] : 0 [45819.745235] .avg_idle : 1000000 [45819.750990] .max_idle_balance_cost : 500000 [45819.758325] cfs_rq[12]:/ [45819.761173] .exec_clock : 0.000000 [45819.767025] .MIN_vruntime : 0.000001 [45819.772889] .min_vruntime : 453812.218502 [45819.779228] .max_vruntime : 0.000001 [45819.785073] .spread : 0.000000 [45819.790925] .spread0 : 173345.581473 [45819.797261] .nr_spread_over : 0 [45819.802419] .nr_running : 0 [45819.807594] .load : 0 [45819.812760] .runnable_load_avg : 0 [45819.817940] .blocked_load_avg : 0 [45819.823113] .tg_load_avg : 0 [45819.828284] .tg_load_contrib : 0 [45819.833463] .tg_runnable_contrib : 0 [45819.838636] .tg->runnable_avg : 0 [45819.843829] .tg->cfs_bandwidth.timer_active: 0 [45819.849002] .throttled : 0 [45819.854182] .throttle_count : 0 [45819.859364] .avg->runnable_avg_sum : 0 [45819.864538] .avg->runnable_avg_period : 46384 [45819.871780] rt_rq[12]:/ [45819.874536] .rt_nr_running : 0 [45819.879711] .rt_throttled : 0 [45819.884897] .rt_time : 0.000000 [45819.890741] .rt_runtime : 950.000000 [45819.898473] runnable tasks: [45819.901617] task PID tree-key switches prio wait-time sum-exec sum-sleep [45819.913673] ---------------------------------------------------------------------------------------------------------- [45819.925643] watchdog/12 69 -11.905099 11457 0 0.000000 208.519660 0.000000 1 / [45819.938121] migration/12 70 0.000000 395 0 0.000000 1175.758898 0.000000 1 / [45819.950566] ksoftirqd/12 71 453799.342388 109058 120 0.000000 2498.165063 0.000000 1 / [45819.963005] kworker/12:0H 73 256.833737 9 100 0.000000 0.045482 0.000000 1 / [45819.975489] khungtaskd 173 453786.083757 383 120 0.000000 1554.259204 0.000000 1 / [45819.987959] kworker/12:1H 898 453799.314878 177343 100 0.000000 3591.142534 0.000000 1 / [45820.000452] systemd-logind 1312 453811.314639 53094 120 0.000000 9986.548040 0.000000 1 / [45820.012914] qmgr 1955 453761.874460 275 120 0.000000 93.915031 0.000000 1 / [45820.025415] ptlrpc_hr01_002 3462 448702.255022 442897 120 0.000000 17812.770245 0.000000 1 / [45820.037869] kiblnd_sd_01_01 3483 453812.257540 44836694 120 0.000000 735133.932191 0.000000 1 / [45820.050328] ptlrpcd_01_06 3513 453799.662867 474025 120 0.000000 17483.181242 0.000000 1 / [45820.062804] ldlm_cb01_000 3540 445037.651678 298532 120 0.000000 17850.232624 0.000000 1 / [45820.075257] ll_evictor 3555 453721.066789 39 120 0.000000 719.430784 0.000000 1 / [45820.087736] mdt01_000 3563 291281.020172 561572 120 0.000000 50904.466458 0.000000 1 / [45820.100205] osp-syn-2-0 3607 448720.578255 219377 120 0.000000 5861.335885 0.000000 1 / [45820.112658] osp-pre-2-0 3608 453800.008837 12405 120 0.000000 647.339089 0.000000 1 / [45820.125118] osp-syn-12-0 3627 448720.817375 218120 120 0.000000 5832.570683 0.000000 1 / [45820.137595] osp-syn-13-0 3629 448731.010808 213854 120 0.000000 5879.158326 0.000000 1 / [45820.150054] mdt01_003 3693 441280.414210 1343844 120 0.000000 126572.482843 0.000000 1 / [45820.162530] mdt_io01_004 3789 446596.565186 39068 120 0.000000 3353.628640 0.000000 1 / [45820.174987] ldlm_bl_10 22100 453786.006144 91271 120 0.000000 2824.165602 0.000000 1 / [45820.187463] mdt01_023 35179 434051.933018 247065 120 0.000000 22186.424017 0.000000 1 / [45820.199924] kworker/12:0 38864 453799.447399 54324 120 0.000000 1114.175734 0.000000 1 / [45820.212386] kworker/12:1 68681 451295.273529 2 120 0.000000 0.069084 0.000000 1 / [45820.226540] cpu#13, 2693.251 MHz [45820.230154] .nr_running : 0 [45820.235321] .load : 0 [45820.240488] .nr_switches : 37845053 [45820.246349] .nr_load_updates : 9358754 [45820.252111] .nr_uninterruptible : -1813 [45820.257671] .next_balance : 4340.485295 [45820.263799] .curr->pid : 0 [45820.268972] .clock : 45820268.811541 [45820.275505] .cpu_load[0] : 0 [45820.280671] .cpu_load[1] : 0 [45820.285853] .cpu_load[2] : 0 [45820.291025] .cpu_load[3] : 0 [45820.296182] .cpu_load[4] : 0 [45820.301357] .avg_idle : 1000000 [45820.307118] .max_idle_balance_cost : 500000 [45820.314455] cfs_rq[13]:/ [45820.317301] .exec_clock : 0.000000 [45820.323161] .MIN_vruntime : 0.000001 [45820.329014] .min_vruntime : 453605.763961 [45820.335344] .max_vruntime : 0.000001 [45820.341190] .spread : 0.000000 [45820.347036] .spread0 : 173138.355567 [45820.353380] .nr_spread_over : 0 [45820.358546] .nr_running : 0 [45820.363722] .load : 0 [45820.368887] .runnable_load_avg : 0 [45820.374067] .blocked_load_avg : 0 [45820.379247] .tg_load_avg : 0 [45820.384428] .tg_load_contrib : 0 [45820.389593] .tg_runnable_contrib : 0 [45820.394764] .tg->runnable_avg : 0 [45820.399930] .tg->cfs_bandwidth.timer_active: 0 [45820.405096] .throttled : 0 [45820.410262] .throttle_count : 0 [45820.415440] .avg->runnable_avg_sum : 0 [45820.420621] .avg->runnable_avg_period : 47361 [45820.427867] rt_rq[13]:/ [45820.430617] .rt_nr_running : 0 [45820.435792] .rt_throttled : 0 [45820.440974] .rt_time : 0.000000 [45820.446821] .rt_runtime : 950.000000 [45820.454549] runnable tasks: [45820.457684] task PID tree-key switches prio wait-time sum-exec sum-sleep [45820.469725] ---------------------------------------------------------------------------------------------------------- [45820.481688] kthreadd 2 453587.297790 1246 120 0.000000 55.545777 0.000000 1 / [45820.494142] watchdog/13 74 -11.903976 11457 0 0.000000 241.946085 0.000000 1 / [45820.506662] migration/13 75 0.000000 387 0 0.000000 1149.001713 0.000000 1 / [45820.519141] ksoftirqd/13 76 453585.261942 106467 120 0.000000 2559.177844 0.000000 1 / [45820.531583] kworker/13:0H 78 336.303433 9 100 0.000000 0.073587 0.000000 1 / [45820.544085] kworker/13:1H 952 453587.184162 171714 100 0.000000 3591.053356 0.000000 1 / [45820.556558] auditd 1068 453583.305694 3130 116 0.000000 235.599842 0.000000 1 / [45820.568998] gdbus 1245 453585.551271 8679 120 0.000000 2048.877190 0.000000 1 / [45820.581460] rsyslogd 1810 453571.256741 95 120 0.000000 16.314909 0.000000 1 / [45820.593932] ptlrpcd_01_05 3512 453587.373840 514962 120 0.000000 17539.633628 0.000000 1 / [45820.606389] ptlrpcd_01_10 3517 453592.147605 473296 120 0.000000 17453.567862 0.000000 1 / [45820.618857] mdt_rdpg01_000 3568 453505.500577 282556 120 0.000000 20784.974009 0.000000 1 / [45820.631321] osp_up1-0 3597 447238.851635 3421802 120 0.000000 204043.131283 0.000000 1 / [45820.643813] mdt_out01_004 3773 450254.063630 927418 120 0.000000 63623.148355 0.000000 1 / [45820.656292] mdt_rdpg01_007 3850 449655.725204 331620 120 0.000000 24721.426582 0.000000 1 / [45820.668767] mdt_rdpg01_011 4801 453585.529411 326122 120 0.000000 24970.510048 0.000000 1 / [45820.681217] ldlm_cn01_015 7588 441206.220441 236017 120 0.000000 13349.741887 0.000000 1 / [45820.693678] ldlm_cb01_004 9899 445536.984880 321526 120 0.000000 18979.673130 0.000000 1 / [45820.706147] ll_mgs_0003 19566 453605.763961 20589 120 0.000000 1400.536936 0.000000 1 / [45820.718600] ldlm_cn01_028 34151 453587.620369 89104 120 0.000000 4468.800107 0.000000 1 / [45820.731020] mdt01_025 35219 441526.871025 551624 120 0.000000 51993.609005 0.000000 1 / [45820.743488] mdt01_028 35264 451173.203198 548528 120 0.000000 52862.057464 0.000000 1 / [45820.755941] ldlm_bl_19 36872 453577.395505 48530 120 0.000000 1328.959268 0.000000 1 / [45820.768414] kworker/13:1 44014 451577.986192 43672 120 0.000000 894.276868 0.000000 1 / [45820.780877] kworker/13:2 70510 453593.763530 627 120 0.000000 15.528214 0.000000 1 / [45820.795002] cpu#14, 2693.251 MHz [45820.798624] .nr_running : 0 [45820.803799] .load : 0 [45820.808974] .nr_switches : 37786759 [45820.814828] .nr_load_updates : 9332084 [45820.820584] .nr_uninterruptible : -1827 [45820.826154] .next_balance : 4340.485865 [45820.832290] .curr->pid : 0 [45820.837458] .clock : 45820836.837435 [45820.843989] .cpu_load[0] : 0 [45820.849163] .cpu_load[1] : 0 [45820.854337] .cpu_load[2] : 0 [45820.859510] .cpu_load[3] : 0 [45820.864678] .cpu_load[4] : 0 [45820.869846] .avg_idle : 1000000 [45820.875623] .max_idle_balance_cost : 500000 [45820.882949] cfs_rq[14]:/ [45820.885802] .exec_clock : 0.000000 [45820.891648] .MIN_vruntime : 0.000001 [45820.897517] .min_vruntime : 447179.863224 [45820.903847] .max_vruntime : 0.000001 [45820.909705] .spread : 0.000000 [45820.915550] .spread0 : 166712.454830 [45820.921903] .nr_spread_over : 0 [45820.927061] .nr_running : 0 [45820.932240] .load : 0 [45820.937412] .runnable_load_avg : 0 [45820.942583] .blocked_load_avg : 0 [45820.947762] .tg_load_avg : 0 [45820.952954] .tg_load_contrib : 0 [45820.958132] .tg_runnable_contrib : 0 [45820.963305] .tg->runnable_avg : 0 [45820.968472] .tg->cfs_bandwidth.timer_active: 0 [45820.973638] .throttled : 0 [45820.978806] .throttle_count : 0 [45820.983979] .avg->runnable_avg_sum : 20 [45820.989256] .avg->runnable_avg_period : 47155 [45820.996498] rt_rq[14]:/ [45820.999248] .rt_nr_running : 0 [45821.004423] .rt_throttled : 0 [45821.009598] .rt_time : 0.000000 [45821.015444] .rt_runtime : 950.000000 [45821.023186] runnable tasks: [45821.026333] task PID tree-key switches prio wait-time sum-exec sum-sleep [45821.038391] ---------------------------------------------------------------------------------------------------------- [45821.050362] watchdog/14 79 -11.906617 11457 0 0.000000 207.687246 0.000000 1 / [45821.062796] migration/14 80 0.000000 487 0 0.000000 1102.097706 0.000000 1 / [45821.075243] ksoftirqd/14 81 447165.110455 106287 120 0.000000 2407.503450 0.000000 1 / [45821.087717] kworker/14:0H 83 355.320567 9 100 0.000000 0.049616 0.000000 1 / [45821.100199] kworker/14:1H 814 447165.081672 173156 100 0.000000 3569.832998 0.000000 1 / [45821.112656] sssd 1104 447163.979780 1728 120 0.000000 151.215025 0.000000 1 / [45821.125099] gdbus 1315 447163.763692 28489 120 0.000000 4101.134925 0.000000 1 / [45821.137552] crond 1932 447158.638494 860 120 0.000000 123.746198 0.000000 1 / [45821.150014] kiblnd_sd_01_02 3484 447179.894890 44832685 120 0.000000 735916.692265 0.000000 1 / [45821.162514] ptlrpcd_01_08 3515 447166.413071 473111 120 0.000000 17495.043692 0.000000 1 / [45821.174962] ptlrpcd_01_13 3520 447167.657703 513942 120 0.000000 17520.510046 0.000000 1 / [45821.187405] jbd2/dm-3-8 3532 447164.803969 1254373 120 0.000000 196510.085278 0.000000 1 / [45821.199841] osp-syn-3-0 3609 442904.061398 215000 120 0.000000 5924.118057 0.000000 1 / [45821.212318] mdt_io01_003 3788 440672.817355 35734 120 0.000000 3158.505044 0.000000 1 / [45821.224792] ldlm_cn01_018 7669 439266.662991 352355 120 0.000000 21608.808068 0.000000 1 / [45821.237251] mdt_out01_015 7834 446808.038399 1062180 120 0.000000 73938.759752 0.000000 1 / [45821.249714] ldlm_cn01_020 7996 434739.076744 263977 120 0.000000 15680.201689 0.000000 1 / [45821.262161] mdt_io01_008 25062 439708.737310 16448 120 0.000000 983.985266 0.000000 1 / [45821.274620] mdt_out01_017 34266 443702.153169 587952 120 0.000000 39244.238646 0.000000 1 / [45821.287060] ldlm_bl_14 36867 447155.615576 48538 120 0.000000 1364.967962 0.000000 1 / [45821.299533] ldlm_bl_17 36870 447152.771123 48541 120 0.000000 1361.629074 0.000000 1 / [45821.311979] kworker/14:1 39374 443693.125976 5 120 0.000000 0.089496 0.000000 1 / [45821.324437] mdt_io01_009 42596 440672.242728 2775 120 0.000000 193.462880 0.000000 1 / [45821.336901] mdt_io01_010 42597 441297.689236 1486 120 0.000000 102.066507 0.000000 1 / [45821.349383] kworker/14:2 47385 447165.155221 34290 120 0.000000 635.504819 0.000000 1 / [45821.361850] kworker/u306:1 62810 447167.843347 9067 120 0.000000 1827.305710 0.000000 1 / [45821.376001] cpu#15, 2693.251 MHz [45821.379622] .nr_running : 0 [45821.384789] .load : 0 [45821.389960] .nr_switches : 37826690 [45821.395805] .nr_load_updates : 9260443 [45821.401561] .nr_uninterruptible : -2441 [45821.407129] .next_balance : 4340.486446 [45821.413280] .curr->pid : 0 [45821.418454] .clock : 45821413.353388 [45821.424987] .cpu_load[0] : 0 [45821.430146] .cpu_load[1] : 0 [45821.435321] .cpu_load[2] : 0 [45821.440486] .cpu_load[3] : 0 [45821.445651] .cpu_load[4] : 0 [45821.450831] .avg_idle : 1000000 [45821.456584] .max_idle_balance_cost : 500000 [45821.463923] cfs_rq[15]:/ [45821.466783] .exec_clock : 0.000000 [45821.472632] .MIN_vruntime : 0.000001 [45821.478491] .min_vruntime : 450933.720486 [45821.484814] .max_vruntime : 0.000001 [45821.490660] .spread : 0.000000 [45821.496512] .spread0 : 170464.546515 [45821.502850] .nr_spread_over : 0 [45821.508028] .nr_running : 0 [45821.513200] .load : 0 [45821.518373] .runnable_load_avg : 0 [45821.523554] .blocked_load_avg : 0 [45821.528727] .tg_load_avg : 0 [45821.533920] .tg_load_contrib : 0 [45821.539098] .tg_runnable_contrib : 0 [45821.544269] .tg->runnable_avg : 0 [45821.549436] .tg->cfs_bandwidth.timer_active: 0 [45821.554611] .throttled : 0 [45821.559784] .throttle_count : 0 [45821.564948] .avg->runnable_avg_sum : 0 [45821.570127] .avg->runnable_avg_period : 47646 [45821.577367] rt_rq[15]:/ [45821.580104] .rt_nr_running : 0 [45821.585330] .rt_throttled : 0 [45821.590511] .rt_time : 0.000000 [45821.596371] .rt_runtime : 950.000000 [45821.604099] runnable tasks: [45821.607237] task PID tree-key switches prio wait-time sum-exec sum-sleep [45821.619295] ---------------------------------------------------------------------------------------------------------- [45821.631260] watchdog/15 84 -11.906360 11457 0 0.000000 211.407405 0.000000 1 / [45821.643720] migration/15 85 0.000000 384 0 0.000000 1133.012504 0.000000 1 / [45821.656190] ksoftirqd/15 86 450921.802584 108446 120 0.000000 2430.434178 0.000000 1 / [45821.668645] kworker/15:0H 88 72.365451 9 100 0.000000 0.047130 0.000000 1 / [45821.681136] kworker/15:1H 932 450921.721040 177887 100 0.000000 3601.655759 0.000000 1 / [45821.693607] rpc.statd 1815 288.839773 13 120 0.000000 34.290273 0.000000 1 / [45821.706067] kiblnd_sd_01_03 3485 450933.153188 44826408 120 0.000000 736413.350389 0.000000 1 / [45821.718506] lnet_discovery 3488 450923.101933 45780 120 0.000000 1066.264808 0.000000 1 / [45821.730982] ptlrpcd_01_07 3514 450922.516679 513385 120 0.000000 17612.539833 0.000000 1 / [45821.743455] ll_cfg_requeue 3556 450922.694002 251 120 0.000000 24.790996 0.000000 1 / [45821.755894] mdt_out01_000 3572 450647.310381 1196674 120 0.000000 82254.393474 0.000000 1 / [45821.768370] osp-syn-6-0 3615 445852.799146 221651 120 0.000000 5883.660454 0.000000 1 / [45821.780834] osp-pre-10-0 3624 450921.828134 13000 120 0.000000 654.083399 0.000000 1 / [45821.793329] ldlm_cn01_004 3791 439307.146563 414662 120 0.000000 25279.778848 0.000000 1 / [45821.805811] ldlm_cn01_007 3798 434434.903533 451181 120 0.000000 27226.663060 0.000000 1 / [45821.818274] mdt_rdpg01_006 3849 448820.948096 255891 120 0.000000 19209.105314 0.000000 1 / [45821.830706] mdt_out01_006 4806 446672.494966 601134 120 0.000000 40975.962816 0.000000 1 / [45821.843161] mdt01_018 8315 430885.726029 664465 120 0.000000 60991.501180 0.000000 1 / [45821.855609] mdt01_019 8316 438207.515630 1731029 120 0.000000 165623.931293 0.000000 1 / [45821.868068] mdt_rdpg01_012 30498 446715.080679 136311 120 0.000000 10004.158935 0.000000 1 / [45821.880498] kworker/15:0 33360 450921.815330 66839 120 0.000000 1813.033635 0.000000 1 / [45821.892946] ldlm_bl_12 36865 450920.066444 48536 120 0.000000 1377.171832 0.000000 1 / [45821.905389] kworker/15:1 55270 375223.523610 2 120 0.000000 0.005708 0.000000 1 / [45821.919534] cpu#16, 2693.251 MHz [45821.923156] .nr_running : 0 [45821.928330] .load : 0 [45821.933514] .nr_switches : 5357550 [45821.939269] .nr_load_updates : 2844105 [45821.945012] .nr_uninterruptible : 297 [45821.950380] .next_balance : 4340.486990 [45821.956523] .curr->pid : 0 [45821.961690] .clock : 45821960.871750 [45821.968215] .cpu_load[0] : 0 [45821.973390] .cpu_load[1] : 0 [45821.978572] .cpu_load[2] : 0 [45821.983745] .cpu_load[3] : 0 [45821.988927] .cpu_load[4] : 0 [45821.994093] .avg_idle : 1000000 [45821.999848] .max_idle_balance_cost : 500000 [45822.007167] cfs_rq[16]:/ [45822.010000] .exec_clock : 0.000000 [45822.015846] .MIN_vruntime : 0.000001 [45822.021696] .min_vruntime : 85623.135260 [45822.027958] .max_vruntime : 0.000001 [45822.033819] .spread : 0.000000 [45822.039673] .spread0 : -194854.252189 [45822.046114] .nr_spread_over : 0 [45822.051296] .nr_running : 0 [45822.056469] .load : 0 [45822.061644] .runnable_load_avg : 0 [45822.066826] .blocked_load_avg : 0 [45822.072005] .tg_load_avg : 0 [45822.077178] .tg_load_contrib : 0 [45822.082343] .tg_runnable_contrib : 0 [45822.087516] .tg->runnable_avg : 0 [45822.092679] .tg->cfs_bandwidth.timer_active: 0 [45822.097857] .throttled : 0 [45822.103029] .throttle_count : 0 [45822.108208] .avg->runnable_avg_sum : 0 [45822.113388] .avg->runnable_avg_period : 46202 [45822.120645] rt_rq[16]:/ [45822.123394] .rt_nr_running : 0 [45822.128569] .rt_throttled : 0 [45822.133736] .rt_time : 0.000000 [45822.139594] .rt_runtime : 950.000000 [45822.147315] runnable tasks: [45822.150452] task PID tree-key switches prio wait-time sum-exec sum-sleep [45822.162503] ---------------------------------------------------------------------------------------------------------- [45822.174482] watchdog/16 89 -11.922052 11457 0 0.000000 173.042647 0.000000 0 / [45822.186932] migration/16 90 0.000000 906 0 0.000000 1098.872854 0.000000 0 / [45822.199388] ksoftirqd/16 91 85611.146295 1814 120 0.000000 43.926439 0.000000 0 / [45822.211822] kworker/16:0 92 5.563870 21 120 0.000000 0.152170 0.000000 0 / [45822.224264] kworker/16:0H 93 492.804897 12 100 0.000000 0.126586 0.000000 0 / [45822.236740] kworker/16:1 239 85611.652451 34997 120 0.000000 858.607354 0.000000 0 / [45822.249218] multipathd 729 0.000000 59 0 0.000000 8.002941 0.000000 0 / [45822.261678] dhclient 1299 85427.955723 89 120 0.000000 31.360715 0.000000 0 / [45822.274145] kworker/16:1H 1962 85502.169050 14367 100 0.000000 302.201872 0.000000 0 / [45822.286597] ntpd 2693 85611.488014 2023 120 0.000000 251.046994 0.000000 0 / [45822.299047] ldlm_cn00_002 3544 81575.591855 213247 120 0.000000 13204.177225 0.000000 0 / [45822.311528] mgs_soaked_noti 3586 85612.360984 290 118 0.000000 51.274246 0.000000 0 / [45822.323981] mdt00_008 3781 85611.196938 574414 120 0.000000 50325.909330 0.000000 0 / [45822.336461] mdt00_014 8311 83247.328708 843375 120 0.000000 69082.750337 0.000000 0 / [45822.348910] mdt_io00_004 9739 82670.946744 25269 120 0.000000 2295.642962 0.000000 0 / [45822.361371] mdt_rdpg00_009 17955 82702.293049 168035 120 0.000000 10322.838874 0.000000 0 / [45822.373836] mdt_out00_013 23165 85074.616835 164420 120 0.000000 11864.658805 0.000000 0 / [45822.386312] sshd 78771 85611.159135 427 120 0.000000 67.882003 0.000000 0 / [45822.400433] cpu#17, 2693.251 MHz [45822.404056] .nr_running : 0 [45822.409231] .load : 0 [45822.414413] .nr_switches : 4161431 [45822.420160] .nr_load_updates : 3302551 [45822.425916] .nr_uninterruptible : 987 [45822.431308] .next_balance : 4340.487471 [45822.437456] .curr->pid : 0 [45822.442636] .clock : 45822437.516924 [45822.449167] .cpu_load[0] : 0 [45822.454345] .cpu_load[1] : 0 [45822.459518] .cpu_load[2] : 0 [45822.464678] .cpu_load[3] : 0 [45822.469837] .cpu_load[4] : 0 [45822.475016] .avg_idle : 1000000 [45822.480771] .max_idle_balance_cost : 500000 [45822.488104] cfs_rq[17]:/ [45822.490973] .exec_clock : 0.000000 [45822.496823] .MIN_vruntime : 0.000001 [45822.502676] .min_vruntime : 723842.171483 [45822.509025] .max_vruntime : 0.000001 [45822.514870] .spread : 0.000000 [45822.520716] .spread0 : 443357.272730 [45822.527055] .nr_spread_over : 0 [45822.532221] .nr_running : 0 [45822.537394] .load : 0 [45822.542568] .runnable_load_avg : 0 [45822.547750] .blocked_load_avg : 0 [45822.552925] .tg_load_avg : 0 [45822.558117] .tg_load_contrib : 0 [45822.563290] .tg_runnable_contrib : 0 [45822.568446] .tg->runnable_avg : 0 [45822.573611] .tg->cfs_bandwidth.timer_active: 0 [45822.578777] .throttled : 0 [45822.583971] .throttle_count : 0 [45822.589192] .avg->runnable_avg_sum : 0 [45822.594368] .avg->runnable_avg_period : 47590 [45822.601601] rt_rq[17]:/ [45822.604352] .rt_nr_running : 0 [45822.609526] .rt_throttled : 0 [45822.614693] .rt_time : 0.000000 [45822.620561] .rt_runtime : 950.000000 [45822.628282] runnable tasks: [45822.631426] task PID tree-key switches prio wait-time sum-exec sum-sleep [45822.643484] ---------------------------------------------------------------------------------------------------------- [45822.655459] watchdog/17 94 -11.925082 11458 0 0.000000 175.354835 0.000000 0 / [45822.667896] migration/17 95 0.000000 15742 0 0.000000 1257.341629 0.000000 0 / [45822.680330] ksoftirqd/17 96 723583.170575 9277 120 0.000000 286.184374 0.000000 0 / [45822.692775] kworker/17:0H 98 867.708834 7 100 0.000000 0.061041 0.000000 0 / [45822.705234] kworker/17:1 187 723830.195949 32907 120 0.000000 667.665883 0.000000 0 / [45822.717694] kworker/17:1H 534 597224.942711 2014 100 0.000000 44.321566 0.000000 0 / [45822.730170] multipathd 866 0.000000 11 0 0.000000 19.887296 0.000000 0 / [45822.742639] multipathd 873 0.000000 2 0 0.000000 0.931216 0.000000 0 / [45822.755108] multipathd 877 0.000000 2 0 0.000000 0.437457 0.000000 0 / [45822.767573] multipathd 883 0.000000 17 0 0.000000 3.457000 0.000000 0 / [45822.780021] multipathd 887 0.000000 1 0 0.000000 0.404937 0.000000 0 / [45822.792476] gmain 1314 4068.362062 3 120 0.000000 0.150220 0.000000 0 / [45822.804923] polkitd 1316 4092.865152 1 120 0.000000 0.132905 0.000000 0 / [45822.817381] JS Sour~ Thread 1318 4118.979974 9 120 0.000000 0.250902 0.000000 0 / [45822.829830] mlx4_en 1307 4042.640965 2 100 0.000000 0.017388 0.000000 0 / [45822.842268] ipoib_wq 1391 4570.363626 2 100 0.000000 0.040948 0.000000 0 / [45822.854731] ipoib_wq 1392 4582.418798 2 100 0.000000 0.056483 0.000000 0 / [45822.867182] nfsiod 1849 6383.665442 2 100 0.000000 0.032134 0.000000 0 / [45822.879663] automount 2173 7903.921928 1 120 0.000000 1.215769 0.000000 0 / [45822.892111] agetty 1998 7715.892012 7 120 0.000000 15.564697 0.000000 0 / [45822.904548] kworker/u308:1 2439 9236.930604 2 100 0.000000 0.105644 0.000000 0 / [45822.917003] ib_fmr(mlx4_0) 3486 16091.215928 2 120 0.000000 0.010175 0.000000 0 / [45822.929475] ldiskfs-rsv-con 3533 16165.026745 2 100 0.000000 0.058223 0.000000 0 / [45822.941933] ldlm_elt 3550 16333.768235 2 120 0.000000 0.013664 0.000000 0 / [45822.954412] mdt_rdpg00_000 3566 656522.673009 236332 120 0.000000 15737.040528 0.000000 0 / [45822.966887] mdt_seqs_0000 3574 16563.861496 2 120 0.000000 0.109323 0.000000 0 / [45822.979344] mdt_seqs_0001 3575 16575.885215 2 120 0.000000 0.023725 0.000000 0 / [45822.991804] qmt_reba_soaked 3593 16746.796411 2 120 0.000000 0.108895 0.000000 0 / [45823.004270] mdt_restriper_0 3594 16758.855253 2 120 0.000000 0.058848 0.000000 0 / [45823.016740] mdt_out00_011 13023 587291.223915 513071 120 0.000000 41208.600481 0.000000 0 / [45823.029199] ldlm_cn00_009 16656 597984.527617 180745 120 0.000000 10596.790742 0.000000 0 / [45823.041666] mdt00_018 22105 602689.033410 973785 120 0.000000 79406.705277 0.000000 0 / [45823.054104] mdt_seqm_0002 43901 702834.294113 25 120 0.000000 1.131573 0.000000 0 / [45823.066561] kworker/17:2 59730 548402.875286 4 120 0.000000 0.082400 0.000000 0 / [45823.080726] cpu#18, 2693.251 MHz [45823.084348] .nr_running : 0 [45823.089521] .load : 0 [45823.094688] .nr_switches : 4203502 [45823.100444] .nr_load_updates : 2710799 [45823.106202] .nr_uninterruptible : 763 [45823.111571] .next_balance : 4340.488151 [45823.117714] .curr->pid : 0 [45823.122887] .clock : 45823121.926523 [45823.129410] .cpu_load[0] : 0 [45823.134587] .cpu_load[1] : 0 [45823.139754] .cpu_load[2] : 0 [45823.144922] .cpu_load[3] : 0 [45823.150089] .cpu_load[4] : 0 [45823.155256] .avg_idle : 1000000 [45823.161016] .max_idle_balance_cost : 500000 [45823.168365] cfs_rq[18]:/ [45823.171204] .exec_clock : 0.000000 [45823.177063] .MIN_vruntime : 0.000001 [45823.182915] .min_vruntime : 150611.421763 [45823.189251] .max_vruntime : 0.000001 [45823.195111] .spread : 0.000000 [45823.200973] .spread0 : -129882.894499 [45823.207400] .nr_spread_over : 0 [45823.212574] .nr_running : 0 [45823.217748] .load : 0 [45823.222916] .runnable_load_avg : 0 [45823.228089] .blocked_load_avg : 0 [45823.233255] .tg_load_avg : 0 [45823.238431] .tg_load_contrib : 0 [45823.243606] .tg_runnable_contrib : 0 [45823.248773] .tg->runnable_avg : 0 [45823.253942] .tg->cfs_bandwidth.timer_active: 0 [45823.259115] .throttled : 0 [45823.264287] .throttle_count : 0 [45823.269467] .avg->runnable_avg_sum : 0 [45823.274626] .avg->runnable_avg_period : 46700 [45823.281870] rt_rq[18]:/ [45823.284606] .rt_nr_running : 0 [45823.289766] .rt_throttled : 0 [45823.294926] .rt_time : 0.000000 [45823.300777] .rt_runtime : 950.000000 [45823.308490] runnable tasks: [45823.311634] task PID tree-key switches prio wait-time sum-exec sum-sleep [45823.323697] ---------------------------------------------------------------------------------------------------------- [45823.335666] watchdog/18 99 -11.926021 11459 0 0.000000 168.049365 0.000000 0 / [45823.348099] migration/18 100 0.000000 2734 0 0.000000 1474.429739 0.000000 0 / [45823.360556] ksoftirqd/18 101 150552.021824 3734 120 0.000000 102.712066 0.000000 0 / [45823.372998] kworker/18:0H 103 2630.635280 9 100 0.000000 0.137850 0.000000 0 / [45823.385430] kworker/18:1 185 150599.464946 27473 120 0.000000 1237.273271 0.000000 0 / [45823.397893] multipathd 728 147300.032294 140 120 0.000000 13.185668 0.000000 0 / [45823.410339] polkitd 1319 1280.672104 3 120 0.000000 0.129086 0.000000 0 / [45823.422813] kworker/18:1H 2396 140832.817696 2438 100 0.000000 52.014073 0.000000 0 / [45823.435234] ptlrpc_hr00_005 3457 142014.138516 268318 120 0.000000 8787.006446 0.000000 0 / [45823.447724] ptlrpcd_00_07 3498 150600.029123 479629 120 0.000000 18904.058610 0.000000 0 / [45823.460197] ldlm_cn00_001 3543 150599.429843 290515 120 0.000000 18253.260321 0.000000 0 / [45823.472639] mdt00_002 3562 142109.786338 1167830 120 0.000000 98451.917945 0.000000 0 / [45823.485078] osp-pre-2-0 3600 150599.471798 9010 120 0.000000 378.164792 0.000000 0 / [45823.497537] osp-pre-3-0 3610 150599.455593 11958 120 0.000000 582.396336 0.000000 0 / [45823.509984] osp-syn-9-0 3621 142022.896974 213144 120 0.000000 5970.227559 0.000000 0 / [45823.522477] ldlm_cn00_007 3797 141494.780007 270889 120 0.000000 16544.697974 0.000000 0 / [45823.534960] kworker/18:0 74257 150282.691259 5 120 0.000000 0.153139 0.000000 0 / [45823.547431] kworker/18:2 78505 150294.702428 5 120 0.000000 0.121683 0.000000 0 / [45823.561566] cpu#19, 2693.251 MHz [45823.565188] .nr_running : 0 [45823.570362] .load : 0 [45823.575549] .nr_switches : 3755306 [45823.581312] .nr_load_updates : 2718061 [45823.587092] .nr_uninterruptible : 489 [45823.592513] .next_balance : 4340.488630 [45823.598671] .curr->pid : 0 [45823.603845] .clock : 45823598.744956 [45823.610377] .cpu_load[0] : 0 [45823.615551] .cpu_load[1] : 0 [45823.620718] .cpu_load[2] : 0 [45823.625894] .cpu_load[3] : 0 [45823.631077] .cpu_load[4] : 0 [45823.636250] .avg_idle : 1000000 [45823.642029] .max_idle_balance_cost : 500000 [45823.649376] cfs_rq[19]:/ [45823.652229] .exec_clock : 0.000000 [45823.658083] .MIN_vruntime : 0.000001 [45823.663937] .min_vruntime : 93996.645755 [45823.670178] .max_vruntime : 0.000001 [45823.676029] .spread : 0.000000 [45823.681873] .spread0 : -186505.336571 [45823.688299] .nr_spread_over : 0 [45823.693466] .nr_running : 0 [45823.698644] .load : 0 [45823.703818] .runnable_load_avg : 0 [45823.709017] .blocked_load_avg : 0 [45823.714198] .tg_load_avg : 0 [45823.719370] .tg_load_contrib : 0 [45823.724536] .tg_runnable_contrib : 0 [45823.729703] .tg->runnable_avg : 0 [45823.734864] .tg->cfs_bandwidth.timer_active: 0 [45823.740036] .throttled : 0 [45823.745207] .throttle_count : 0 [45823.750379] .avg->runnable_avg_sum : 0 [45823.755558] .avg->runnable_avg_period : 47037 [45823.762810] rt_rq[19]:/ [45823.765560] .rt_nr_running : 0 [45823.770741] .rt_throttled : 0 [45823.775907] .rt_time : 0.000000 [45823.781759] .rt_runtime : 950.000000 [45823.789479] runnable tasks: [45823.792614] task PID tree-key switches prio wait-time sum-exec sum-sleep [45823.804658] ---------------------------------------------------------------------------------------------------------- [45823.816623] watchdog/19 104 -11.927355 11459 0 0.000000 172.239705 0.000000 0 / [45823.829070] migration/19 105 0.000000 1109 0 0.000000 1112.514509 0.000000 0 / [45823.841542] ksoftirqd/19 106 93794.170844 1654 120 0.000000 44.722644 0.000000 0 / [45823.854010] kworker/19:0H 108 2899.287123 9 100 0.000000 0.068827 0.000000 0 / [45823.866463] kworker/19:1 240 93984.665501 28694 120 0.000000 709.182019 0.000000 0 / [45823.878916] poll_mpt2sas0_s 524 202.328448 2 100 0.000000 0.069062 0.000000 0 / [45823.891402] mdt00_001 3561 91232.445762 1084217 120 0.000000 91520.124796 0.000000 0 / [45823.903890] mdt_io00_000 3580 90898.989419 20830 120 0.000000 1854.411435 0.000000 0 / [45823.916380] ldlm_cn00_003 3777 90937.489266 308816 120 0.000000 18983.231769 0.000000 0 / [45823.928846] mdt_rdpg00_004 3845 91326.635002 243636 120 0.000000 17172.328989 0.000000 0 / [45823.941310] kworker/19:1H 5126 90648.803741 1510 100 0.000000 42.762124 0.000000 0 / [45823.953762] kworker/19:2 15739 22268.478722 2 120 0.000000 0.029413 0.000000 0 / [45823.966227] ldlm_cn00_017 24512 92960.708565 194837 120 0.000000 10203.873523 0.000000 0 / [45823.978712] ldlm_cn00_018 24513 88875.382923 121640 120 0.000000 6873.423128 0.000000 0 / [45823.991167] mdt_out00_016 25139 88864.444730 287117 120 0.000000 23223.281705 0.000000 0 / [45824.005322] cpu#20, 2693.251 MHz [45824.008938] .nr_running : 0 [45824.014106] .load : 0 [45824.019282] .nr_switches : 5008546 [45824.025037] .nr_load_updates : 2709259 [45824.030799] .nr_uninterruptible : 829 [45824.036167] .next_balance : 4340.489075 [45824.042304] .curr->pid : 0 [45824.047479] .clock : 45824046.968456 [45824.054018] .cpu_load[0] : 0 [45824.059190] .cpu_load[1] : 0 [45824.064342] .cpu_load[2] : 0 [45824.069501] .cpu_load[3] : 0 [45824.074661] .cpu_load[4] : 0 [45824.079820] .avg_idle : 1000000 [45824.085569] .max_idle_balance_cost : 500000 [45824.092904] cfs_rq[20]:/ [45824.095762] .exec_clock : 0.000000 [45824.101622] .MIN_vruntime : 0.000001 [45824.107475] .min_vruntime : 109392.366560 [45824.113812] .max_vruntime : 0.000001 [45824.119650] .spread : 0.000000 [45824.125502] .spread0 : -171116.906421 [45824.131929] .nr_spread_over : 0 [45824.137105] .nr_running : 0 [45824.142275] .load : 0 [45824.147453] .runnable_load_avg : 0 [45824.152626] .blocked_load_avg : 0 [45824.157799] .tg_load_avg : 0 [45824.162967] .tg_load_contrib : 0 [45824.168133] .tg_runnable_contrib : 0 [45824.173304] .tg->runnable_avg : 0 [45824.178476] .tg->cfs_bandwidth.timer_active: 0 [45824.183640] .throttled : 0 [45824.188812] .throttle_count : 0 [45824.193974] .avg->runnable_avg_sum : 0 [45824.199148] .avg->runnable_avg_period : 46997 [45824.206380] rt_rq[20]:/ [45824.209128] .rt_nr_running : 0 [45824.214308] .rt_throttled : 0 [45824.219474] .rt_time : 0.000000 [45824.225326] .rt_runtime : 950.000000 [45824.233040] runnable tasks: [45824.236174] task PID tree-key switches prio wait-time sum-exec sum-sleep [45824.248229] ---------------------------------------------------------------------------------------------------------- [45824.260194] watchdog/20 109 -11.932100 11458 0 0.000000 174.972063 0.000000 0 / [45824.272655] migration/20 110 0.000000 2150 0 0.000000 1113.496276 0.000000 0 / [45824.285088] ksoftirqd/20 111 104941.385014 1489 120 0.000000 37.935130 0.000000 0 / [45824.297564] kworker/20:0H 113 746.230085 9 100 0.000000 0.055036 0.000000 0 / [45824.309999] kworker/20:1 241 109368.349121 27560 120 0.000000 657.444127 0.000000 0 / [45824.322497] scsi_tmf_2 485 165.684264 2 100 0.000000 0.013397 0.000000 0 / [45824.334955] scsi_tmf_3 487 189.692605 2 100 0.000000 0.004328 0.000000 0 / [45824.347411] scsi_tmf_4 489 213.700806 2 100 0.000000 0.004446 0.000000 0 / [45824.359890] scsi_eh_5 490 310.258881 22 120 0.000000 0.485780 0.000000 0 / [45824.372328] scsi_tmf_5 491 237.708812 2 100 0.000000 0.004426 0.000000 0 / [45824.384738] scsi_tmf_6 493 261.716854 2 100 0.000000 0.004415 0.000000 0 / [45824.397170] scsi_tmf_7 495 285.725706 2 100 0.000000 0.004670 0.000000 0 / [45824.409636] mlx4_ib_mcg 546 358.377679 2 100 0.000000 0.010098 0.000000 0 / [45824.422118] rpciod 689 407.965537 2 100 0.000000 0.053759 0.000000 0 / [45824.434605] kdmflush 860 637.815637 2 100 0.000000 0.005825 0.000000 0 / [45824.447059] bioset 864 649.820584 2 100 0.000000 0.005472 0.000000 0 / [45824.459516] kdmflush 870 686.080019 2 100 0.000000 0.005392 0.000000 0 / [45824.471986] bioset 871 698.084251 2 100 0.000000 0.004651 0.000000 0 / [45824.484454] kdmflush 874 722.095247 2 100 0.000000 0.005215 0.000000 0 / [45824.496908] bioset 875 734.099165 2 100 0.000000 0.004376 0.000000 0 / [45824.509354] kdmwork-253:2 876 884.434158 99 120 0.000000 1.904481 0.000000 0 / [45824.521805] kworker/20:1H 910 104160.453992 1372 100 0.000000 34.255941 0.000000 0 / [45824.534268] ldlm_cb00_000 3538 104107.493181 75382 120 0.000000 4897.614960 0.000000 0 / [45824.546754] mdt_io00_002 3582 104556.499009 4515 120 0.000000 407.907726 0.000000 0 / [45824.559251] ldlm_cn00_014 17023 107741.698130 197222 120 0.000000 11808.216922 0.000000 0 / [45824.571719] mdt00_016 22102 109380.408533 523262 120 0.000000 41893.366130 0.000000 0 / [45824.584173] kworker/20:2 25456 38447.074096 2 120 0.000000 0.004707 0.000000 0 / [45824.598338] cpu#21, 2693.251 MHz [45824.601968] .nr_running : 0 [45824.607150] .load : 0 [45824.612330] .nr_switches : 3938740 [45824.618076] .nr_load_updates : 2795050 [45824.623823] .nr_uninterruptible : 278 [45824.629196] .next_balance : 4340.489668 [45824.635345] .curr->pid : 0 [45824.640517] .clock : 45824639.994719 [45824.647065] .cpu_load[0] : 0 [45824.652244] .cpu_load[1] : 0 [45824.657423] .cpu_load[2] : 0 [45824.662600] .cpu_load[3] : 0 [45824.667781] .cpu_load[4] : 0 [45824.672955] .avg_idle : 1000000 [45824.678712] .max_idle_balance_cost : 500000 [45824.686064] cfs_rq[21]:/ [45824.688912] .exec_clock : 0.000000 [45824.694752] .MIN_vruntime : 0.000001 [45824.700612] .min_vruntime : 101365.779604 [45824.706953] .max_vruntime : 0.000001 [45824.712809] .spread : 0.000000 [45824.718662] .spread0 : -179152.199239 [45824.725094] .nr_spread_over : 0 [45824.730288] .nr_running : 0 [45824.735459] .load : 0 [45824.740640] .runnable_load_avg : 0 [45824.745806] .blocked_load_avg : 0 [45824.750981] .tg_load_avg : 0 [45824.756149] .tg_load_contrib : 0 [45824.761323] .tg_runnable_contrib : 0 [45824.766496] .tg->runnable_avg : 0 [45824.771670] .tg->cfs_bandwidth.timer_active: 0 [45824.776837] .throttled : 0 [45824.782004] .throttle_count : 0 [45824.787184] .avg->runnable_avg_sum : 0 [45824.792348] .avg->runnable_avg_period : 47969 [45824.799594] rt_rq[21]:/ [45824.802351] .rt_nr_running : 0 [45824.807523] .rt_throttled : 0 [45824.812679] .rt_time : 0.000000 [45824.818525] .rt_runtime : 950.000000 [45824.826231] runnable tasks: [45824.829361] task PID tree-key switches prio wait-time sum-exec sum-sleep [45824.841420] ---------------------------------------------------------------------------------------------------------- [45824.853379] watchdog/21 114 -11.933267 11458 0 0.000000 175.528989 0.000000 0 / [45824.865846] migration/21 115 0.000000 1676 0 0.000000 1110.648872 0.000000 0 / [45824.878284] ksoftirqd/21 116 101266.134850 1797 120 0.000000 43.215605 0.000000 0 / [45824.890739] kworker/21:0H 118 638.764999 9 100 0.000000 0.109264 0.000000 0 / [45824.903198] kworker/21:1H 1864 100803.665083 3224 100 0.000000 88.191221 0.000000 0 / [45824.915687] ldlm_cn00_000 3542 101353.839005 284720 120 0.000000 17206.082968 0.000000 0 / [45824.928170] mdt_rdpg00_007 4642 95192.786051 280719 120 0.000000 17931.499742 0.000000 0 / [45824.940639] ldlm_cb00_005 13699 94815.419920 66972 120 0.000000 3991.744681 0.000000 0 / [45824.953099] ldlm_cn00_010 16657 93787.827633 135343 120 0.000000 8008.918013 0.000000 0 / [45824.965554] mdt_io00_005 30182 95129.455603 803 120 0.000000 60.490677 0.000000 0 / [45824.978013] kworker/21:0 48578 101353.863553 13852 120 0.000000 326.377800 0.000000 0 / [45824.990488] kworker/21:1 61122 93738.861226 3 120 0.000000 0.031791 0.000000 0 / [45825.004614] cpu#22, 2693.251 MHz [45825.008235] .nr_running : 0 [45825.013409] .load : 0 [45825.018591] .nr_switches : 13073412 [45825.024428] .nr_load_updates : 3162761 [45825.030176] .nr_uninterruptible : 462 [45825.035545] .next_balance : 4340.490075 [45825.041690] .curr->pid : 0 [45825.046870] .clock : 45825046.014326 [45825.053387] .cpu_load[0] : 0 [45825.058568] .cpu_load[1] : 0 [45825.063743] .cpu_load[2] : 0 [45825.068917] .cpu_load[3] : 0 [45825.074082] .cpu_load[4] : 0 [45825.079253] .avg_idle : 1000000 [45825.085001] .max_idle_balance_cost : 500000 [45825.092337] cfs_rq[22]:/ [45825.095183] .exec_clock : 0.000000 [45825.101054] .MIN_vruntime : 0.000001 [45825.106906] .min_vruntime : 139575.146940 [45825.113241] .max_vruntime : 0.000001 [45825.119095] .spread : 0.000000 [45825.124956] .spread0 : -140949.571625 [45825.131394] .nr_spread_over : 0 [45825.136565] .nr_running : 0 [45825.141739] .load : 0 [45825.146913] .runnable_load_avg : 0 [45825.152078] .blocked_load_avg : 0 [45825.157251] .tg_load_avg : 0 [45825.162407] .tg_load_contrib : 0 [45825.167585] .tg_runnable_contrib : 0 [45825.172756] .tg->runnable_avg : 0 [45825.177936] .tg->cfs_bandwidth.timer_active: 0 [45825.183107] .throttled : 0 [45825.188302] .throttle_count : 0 [45825.193528] .avg->runnable_avg_sum : 0 [45825.198712] .avg->runnable_avg_period : 47390 [45825.205954] rt_rq[22]:/ [45825.208696] .rt_nr_running : 0 [45825.213872] .rt_throttled : 0 [45825.219063] .rt_time : 0.000000 [45825.224916] .rt_runtime : 950.000000 [45825.232631] runnable tasks: [45825.235772] task PID tree-key switches prio wait-time sum-exec sum-sleep [45825.247836] ---------------------------------------------------------------------------------------------------------- [45825.259802] watchdog/22 119 -11.932905 11458 0 0.000000 169.054567 0.000000 0 / [45825.272263] migration/22 120 0.000000 1317 0 0.000000 1102.272583 0.000000 0 / [45825.284697] ksoftirqd/22 121 139563.191102 3294 120 0.000000 73.274936 0.000000 0 / [45825.297133] kworker/22:0H 123 2190.497636 9 100 0.000000 0.087792 0.000000 0 / [45825.309606] kworker/22:1 244 139547.013520 24156 120 0.000000 741.165251 0.000000 0 / [45825.322076] nfsv4.0-svc 1874 1384.130294 4 120 0.000000 0.278793 0.000000 0 / [45825.334550] osp-syn-1-0 3605 138559.414440 211074 120 0.000000 5901.654187 0.000000 0 / [45825.347032] kworker/22:1H 5119 136627.141554 5742 100 0.000000 101.465435 0.000000 0 / [45825.359492] ldlm_cb00_004 9905 137701.545791 64669 120 0.000000 4315.121863 0.000000 0 / [45825.371958] mdt_out00_005 12980 134610.547303 369131 120 0.000000 28588.757312 0.000000 0 / [45825.384416] mdt_out00_007 12982 134738.334564 357554 120 0.000000 26998.804957 0.000000 0 / [45825.396854] ldlm_cn00_011 16696 135282.278829 139308 120 0.000000 8063.298399 0.000000 0 / [45825.409303] kworker/22:0 16744 25080.980344 3 120 0.000000 0.060046 0.000000 0 / [45825.421769] kworker/u304:0 71789 139563.177945 14366 120 0.000000 756.658869 0.000000 0 / [45825.434230] kworker/u304:1 77030 139563.585092 2932 120 0.000000 130.298986 0.000000 0 / [45825.448367] cpu#23, 2693.251 MHz [45825.451989] .nr_running : 0 [45825.457165] .load : 0 [45825.462333] .nr_switches : 31647295 [45825.468186] .nr_load_updates : 4074604 [45825.473941] .nr_uninterruptible : -236 [45825.479400] .next_balance : 4340.490518 [45825.485535] .curr->pid : 0 [45825.490715] .clock : 45825490.034959 [45825.497252] .cpu_load[0] : 0 [45825.502418] .cpu_load[1] : 0 [45825.507600] .cpu_load[2] : 0 [45825.512761] .cpu_load[3] : 0 [45825.517934] .cpu_load[4] : 0 [45825.523099] .avg_idle : 1000000 [45825.528854] .max_idle_balance_cost : 500000 [45825.536169] cfs_rq[23]:/ [45825.539014] .exec_clock : 0.000000 [45825.544858] .MIN_vruntime : 0.000001 [45825.550710] .min_vruntime : 242165.517241 [45825.557048] .max_vruntime : 0.000001 [45825.562907] .spread : 0.000000 [45825.568759] .spread0 : -38366.431534 [45825.575101] .nr_spread_over : 0 [45825.580268] .nr_running : 0 [45825.585442] .load : 0 [45825.590608] .runnable_load_avg : 0 [45825.595790] .blocked_load_avg : 0 [45825.600956] .tg_load_avg : 0 [45825.606121] .tg_load_contrib : 0 [45825.611286] .tg_runnable_contrib : 0 [45825.616451] .tg->runnable_avg : 0 [45825.621618] .tg->cfs_bandwidth.timer_active: 0 [45825.626789] .throttled : 0 [45825.631969] .throttle_count : 0 [45825.637153] .avg->runnable_avg_sum : 0 [45825.642326] .avg->runnable_avg_period : 47769 [45825.649553] rt_rq[23]:/ [45825.652304] .rt_nr_running : 0 [45825.657478] .rt_throttled : 0 [45825.662651] .rt_time : 0.000000 [45825.668495] .rt_runtime : 950.000000 [45825.676208] runnable tasks: [45825.679336] task PID tree-key switches prio wait-time sum-exec sum-sleep [45825.691407] ---------------------------------------------------------------------------------------------------------- [45825.703373] watchdog/23 124 -11.925547 11459 0 0.000000 170.820984 0.000000 0 / [45825.715826] migration/23 125 0.000000 1520 0 0.000000 1106.771132 0.000000 0 / [45825.728288] ksoftirqd/23 126 242153.540953 1570 120 0.000000 44.379886 0.000000 0 / [45825.740730] kworker/23:0H 128 451.962727 9 100 0.000000 0.088347 0.000000 0 / [45825.753174] kworker/23:1 243 242153.656024 18455 120 0.000000 495.878803 0.000000 0 / [45825.765650] kworker/23:1H 1538 242151.772721 4642 100 0.000000 116.090712 0.000000 0 / [45825.778112] xinetd 1806 558.056642 17 120 0.000000 21.350830 0.000000 0 / [45825.790586] munged 1808 562.215407 13 120 0.000000 2.888879 0.000000 0 / [45825.803047] automount 2165 1511.256678 6 120 0.000000 3.558213 0.000000 0 / [45825.815513] automount 2168 1512.399848 4 120 0.000000 1.408559 0.000000 0 / [45825.827996] ib_fmr(mlx4_0) 3487 228265.841766 38 120 0.000000 0.738956 0.000000 0 / [45825.840456] osp-syn-5-0 3613 235328.827048 210864 120 0.000000 5758.512986 0.000000 0 / [45825.852940] mdt00_006 3776 242153.780459 1094036 120 0.000000 88539.750173 0.000000 0 / [45825.865387] ldlm_bl_05 5124 242137.733135 212035 120 0.000000 7494.619358 0.000000 0 / [45825.877839] mdt00_012 5290 178971.648057 606364 120 0.000000 50301.987833 0.000000 0 / [45825.890323] kworker/u305:0 42955 205100.178299 9 120 0.000000 0.474621 0.000000 0 / [45825.902785] kworker/23:2 53244 166679.990565 2 120 0.000000 0.083859 0.000000 0 / [45825.916916] cpu#24, 2693.251 MHz [45825.920533] .nr_running : 0 [45825.925707] .load : 0 [45825.930880] .nr_switches : 28994279 [45825.936726] .nr_load_updates : 5445880 [45825.942475] .nr_uninterruptible : 2060 [45825.947946] .next_balance : 4340.490987 [45825.954114] .curr->pid : 0 [45825.959264] .clock : 45825954.176920 [45825.965795] .cpu_load[0] : 0 [45825.970977] .cpu_load[1] : 0 [45825.976143] .cpu_load[2] : 0 [45825.981323] .cpu_load[3] : 0 [45825.986495] .cpu_load[4] : 0 [45825.991669] .avg_idle : 1000000 [45825.997422] .max_idle_balance_cost : 500000 [45826.004786] cfs_rq[24]:/ [45826.007632] .exec_clock : 0.000000 [45826.013493] .MIN_vruntime : 0.000001 [45826.019332] .min_vruntime : 284801.123000 [45826.025670] .max_vruntime : 0.000001 [45826.031516] .spread : 0.000000 [45826.037361] .spread0 : 4261.524014 [45826.043497] .nr_spread_over : 0 [45826.048669] .nr_running : 0 [45826.053847] .load : 0 [45826.059019] .runnable_load_avg : 0 [45826.064191] .blocked_load_avg : 0 [45826.069348] .tg_load_avg : 0 [45826.074508] .tg_load_contrib : 0 [45826.079685] .tg_runnable_contrib : 0 [45826.084864] .tg->runnable_avg : 0 [45826.090036] .tg->cfs_bandwidth.timer_active: 0 [45826.095207] .throttled : 0 [45826.100388] .throttle_count : 0 [45826.105561] .avg->runnable_avg_sum : 0 [45826.110742] .avg->runnable_avg_period : 47053 [45826.117987] rt_rq[24]:/ [45826.120730] .rt_nr_running : 0 [45826.125902] .rt_throttled : 0 [45826.131069] .rt_time : 0.000000 [45826.136921] .rt_runtime : 950.000000 [45826.144633] runnable tasks: [45826.147762] task PID tree-key switches prio wait-time sum-exec sum-sleep [45826.159812] ---------------------------------------------------------------------------------------------------------- [45826.171778] watchdog/24 129 -11.916748 11458 0 0.000000 179.520306 0.000000 1 / [45826.184229] migration/24 130 0.000000 890 0 0.000000 1095.597021 0.000000 1 / [45826.196752] ksoftirqd/24 131 281476.577651 3598 120 0.000000 98.877164 0.000000 1 / [45826.209214] kworker/24:0H 133 8968.078782 9 100 0.000000 0.081689 0.000000 1 / [45826.221694] kworker/24:1 245 284789.140504 41205 120 0.000000 815.000557 0.000000 1 / [45826.234187] JS GC Helper 1317 584.954555 2 120 0.000000 0.163781 0.000000 1 / [45826.246636] mlx5_ib_sigerr_ 1363 609.160269 2 100 0.000000 0.055555 0.000000 1 / [45826.259110] rdma_cm 1476 675.730395 2 100 0.000000 0.011824 0.000000 1 / [45826.271607] ldlm_cn01_011 4919 284787.013166 425790 120 0.000000 25550.165213 0.000000 1 / [45826.284072] mdt_out01_008 4920 284660.576980 1320243 120 0.000000 91409.742355 0.000000 1 / [45826.296536] ldlm_cn01_013 5599 274071.608064 410304 120 0.000000 24222.060807 0.000000 1 / [45826.308996] kworker/24:1H 6411 278632.756101 4371 100 0.000000 73.385842 0.000000 1 / [45826.321469] mdt_out01_013 7672 279988.820520 830567 120 0.000000 53924.732715 0.000000 1 / [45826.333935] mdt_io01_006 9897 278106.220861 40851 120 0.000000 3241.938535 0.000000 1 / [45826.346404] ldlm_cb01_009 22097 276904.340272 124100 120 0.000000 6456.464474 0.000000 1 / [45826.358859] ldlm_cn01_025 34148 245634.252724 98299 120 0.000000 5076.444337 0.000000 1 / [45826.371322] mdt01_027 35261 284789.759786 598875 120 0.000000 57519.271005 0.000000 1 / [45826.383770] kworker/24:2 47701 208969.984682 5 120 0.000000 0.126995 0.000000 1 / [45826.397909] cpu#25, 2693.251 MHz [45826.401537] .nr_running : 0 [45826.406710] .load : 0 [45826.411888] .nr_switches : 26705406 [45826.417766] .nr_load_updates : 5683595 [45826.423522] .nr_uninterruptible : 2059 [45826.428987] .next_balance : 4340.491468 [45826.435143] .curr->pid : 0 [45826.440311] .clock : 45826440.088676 [45826.446834] .cpu_load[0] : 0 [45826.452014] .cpu_load[1] : 0 [45826.457199] .cpu_load[2] : 0 [45826.462366] .cpu_load[3] : 0 [45826.467547] .cpu_load[4] : 0 [45826.472720] .avg_idle : 1000000 [45826.478480] .max_idle_balance_cost : 500000 [45826.485813] cfs_rq[25]:/ [45826.488651] .exec_clock : 0.000000 [45826.494490] .MIN_vruntime : 0.000001 [45826.500342] .min_vruntime : 959938.407428 [45826.506687] .max_vruntime : 0.000001 [45826.512523] .spread : 0.000000 [45826.518376] .spread0 : 679388.702909 [45826.524705] .nr_spread_over : 0 [45826.529885] .nr_running : 0 [45826.535057] .load : 0 [45826.540230] .runnable_load_avg : 0 [45826.545392] .blocked_load_avg : 0 [45826.550574] .tg_load_avg : 0 [45826.555747] .tg_load_contrib : 0 [45826.560903] .tg_runnable_contrib : 0 [45826.566071] .tg->runnable_avg : 0 [45826.571236] .tg->cfs_bandwidth.timer_active: 0 [45826.576407] .throttled : 0 [45826.581578] .throttle_count : 0 [45826.586759] .avg->runnable_avg_sum : 0 [45826.591938] .avg->runnable_avg_period : 47113 [45826.599187] rt_rq[25]:/ [45826.601928] .rt_nr_running : 0 [45826.607122] .rt_throttled : 0 [45826.612298] .rt_time : 0.000000 [45826.618151] .rt_runtime : 950.000000 [45826.625871] runnable tasks: [45826.629005] task PID tree-key switches prio wait-time sum-exec sum-sleep [45826.641056] ---------------------------------------------------------------------------------------------------------- [45826.653031] watchdog/25 134 -11.916161 11458 0 0.000000 216.783000 0.000000 1 / [45826.665485] migration/25 135 0.000000 18794 0 0.000000 1475.720278 0.000000 1 / [45826.677931] ksoftirqd/25 136 959454.123588 12124 120 0.000000 464.105863 0.000000 1 / [45826.690385] kworker/25:0H 138 4673.194358 9 100 0.000000 0.064226 0.000000 1 / [45826.702835] netns 172 223.182203 2 100 0.000000 0.019104 0.000000 1 / [45826.715283] writeback 174 247.208175 2 100 0.000000 0.014042 0.000000 1 / [45826.727754] kintegrityd 175 259.219987 2 100 0.000000 0.012929 0.000000 1 / [45826.740205] bioset 176 271.231609 2 100 0.000000 0.012719 0.000000 1 / [45826.752638] bioset 177 283.242592 2 100 0.000000 0.012074 0.000000 1 / [45826.765068] bioset 178 295.254169 2 100 0.000000 0.012622 0.000000 1 / [45826.777514] kblockd 179 307.266111 2 100 0.000000 0.013013 0.000000 1 / [45826.789954] md 181 331.287901 2 100 0.000000 0.012856 0.000000 1 / [45826.802407] edac-poller 182 343.300238 2 100 0.000000 0.013481 0.000000 1 / [45826.814827] watchdogd 183 355.312188 2 100 0.000000 0.013090 0.000000 1 / [45826.827263] ksmd 194 487.692067 2 125 0.000000 0.015960 0.000000 1 / [45826.839696] crypto 196 511.911103 2 100 0.000000 0.020661 0.000000 1 / [45826.852152] kthrotld 204 608.357879 2 100 0.000000 0.014429 0.000000 1 / [45826.864588] kmpath_rdacd 209 646.910127 2 100 0.000000 0.022012 0.000000 1 / [45826.877044] kaluad 210 658.927961 2 100 0.000000 0.018836 0.000000 1 / [45826.889489] kpsmoused 213 682.965093 2 100 0.000000 0.020852 0.000000 1 / [45826.901935] ipv6_addrconf 214 694.978090 2 100 0.000000 0.014280 0.000000 1 / [45826.914421] deferwq 228 815.409209 2 100 0.000000 0.026757 0.000000 1 / [45826.926889] kworker/25:1 252 959926.455044 47843 120 0.000000 1005.731284 0.000000 1 / [45826.939347] mlx4 502 1759.444440 2 100 0.000000 0.037124 0.000000 1 / [45826.951800] mlx4_health 503 1771.491320 2 100 0.000000 0.048161 0.000000 1 / [45826.964257] ib-comp-wq 537 2597.666153 2 100 0.000000 0.046034 0.000000 1 / [45826.976720] kworker/u309:0 540 2633.688360 3 100 0.000000 0.012623 0.000000 1 / [45826.989183] ib-comp-unb-wq 541 2645.696003 2 100 0.000000 0.008733 0.000000 1 / [45827.001644] ib_mcast 542 2657.705118 2 100 0.000000 0.010186 0.000000 1 / [45827.014101] ib_nl_sa_wq 543 2669.714226 2 100 0.000000 0.010120 0.000000 1 / [45827.026572] mlx4_ib 545 2693.750838 2 100 0.000000 0.031103 0.000000 1 / [45827.039054] ib_mad1 547 2705.821082 2 100 0.000000 0.072233 0.000000 1 / [45827.051513] to_fifo 548 2717.882696 2 100 0.000000 0.063689 0.000000 1 / [45827.063959] ib_mad2 549 2729.952822 2 100 0.000000 0.072234 0.000000 1 / [45827.076424] to_fifo 550 2742.012687 2 100 0.000000 0.070086 0.000000 1 / [45827.088883] ext4-rsv-conver 590 2896.890915 2 100 0.000000 0.024910 0.000000 1 / [45827.101325] xprtiod 690 3306.271032 2 100 0.000000 0.010523 0.000000 1 / [45827.113794] lvmetad 702 3353.126510 4 120 0.000000 5.578796 0.000000 1 / [45827.126240] kmpathd 716 3459.326968 2 100 0.000000 0.015914 0.000000 1 / [45827.138691] kmpath_handlerd 717 3471.338925 2 100 0.000000 0.012906 0.000000 1 / [45827.151165] kworker/25:1H 859 959237.289758 3876 100 0.000000 62.586960 0.000000 1 / [45827.163645] gssproxy 1187 5957.768856 1 120 0.000000 0.051768 0.000000 1 / [45827.176122] gssproxy 1189 5969.782466 1 120 0.000000 0.013616 0.000000 1 / [45827.188611] gssproxy 1190 5981.807374 1 120 0.000000 0.024914 0.000000 1 / [45827.201120] ldlm_cn01_006 3795 819880.797456 410031 120 0.000000 24782.815475 0.000000 1 / [45827.213602] ldlm_cn01_019 7829 959237.419168 391638 120 0.000000 24371.228492 0.000000 1 / [45827.226058] ldlm_cn01_029 34265 819558.105122 221619 120 0.000000 12630.608872 0.000000 1 / [45827.238530] kworker/25:2 76446 934277.640691 2 120 0.000000 0.069331 0.000000 1 / [45827.252693] cpu#26, 2693.251 MHz [45827.256310] .nr_running : 0 [45827.261478] .load : 0 [45827.266653] .nr_switches : 28042848 [45827.272497] .nr_load_updates : 5335995 [45827.278252] .nr_uninterruptible : 2014 [45827.283718] .next_balance : 4340.492323 [45827.289860] .curr->pid : 0 [45827.295040] .clock : 45827294.121304 [45827.301578] .cpu_load[0] : 0 [45827.306753] .cpu_load[1] : 0 [45827.311926] .cpu_load[2] : 0 [45827.317092] .cpu_load[3] : 0 [45827.322266] .cpu_load[4] : 0 [45827.327438] .avg_idle : 1000000 [45827.333198] .max_idle_balance_cost : 500000 [45827.340538] cfs_rq[26]:/ [45827.343391] .exec_clock : 0.000000 [45827.349229] .MIN_vruntime : 0.000001 [45827.355088] .min_vruntime : 353909.079397 [45827.361434] .max_vruntime : 0.000001 [45827.367285] .spread : 0.000000 [45827.373122] .spread0 : 73344.641444 [45827.379347] .nr_spread_over : 0 [45827.384513] .nr_running : 0 [45827.389688] .load : 0 [45827.394880] .runnable_load_avg : 0 [45827.400054] .blocked_load_avg : 0 [45827.405242] .tg_load_avg : 0 [45827.410416] .tg_load_contrib : 0 [45827.415600] .tg_runnable_contrib : 0 [45827.420794] .tg->runnable_avg : 0 [45827.425976] .tg->cfs_bandwidth.timer_active: 0 [45827.431152] .throttled : 0 [45827.436326] .throttle_count : 0 [45827.441502] .avg->runnable_avg_sum : 0 [45827.446677] .avg->runnable_avg_period : 47206 [45827.453935] rt_rq[26]:/ [45827.456677] .rt_nr_running : 0 [45827.461850] .rt_throttled : 0 [45827.467024] .rt_time : 0.000000 [45827.472876] .rt_runtime : 950.000000 [45827.480603] runnable tasks: [45827.483747] task PID tree-key switches prio wait-time sum-exec sum-sleep [45827.495805] ---------------------------------------------------------------------------------------------------------- [45827.507778] watchdog/26 139 -11.915759 11459 0 0.000000 183.608396 0.000000 1 / [45827.520218] migration/26 140 0.000000 2175 0 0.000000 1115.393482 0.000000 1 / [45827.532705] ksoftirqd/26 141 353187.872338 6459 120 0.000000 173.184373 0.000000 1 / [45827.545148] kworker/26:0H 143 1132.841694 9 100 0.000000 0.060453 0.000000 1 / [45827.557671] scsi_eh_0 479 289.826585 2 120 0.000000 0.007071 0.000000 1 / [45827.570128] scsi_tmf_0 480 301.832982 2 100 0.000000 0.012005 0.000000 1 / [45827.582590] scsi_eh_1 481 413.242349 7 120 0.000000 0.228511 0.000000 1 / [45827.595035] scsi_tmf_1 482 325.844756 2 100 0.000000 0.006434 0.000000 1 / [45827.607480] scsi_wq_1 483 337.877663 2 100 0.000000 0.034068 0.000000 1 / [45827.619929] kworker/26:1H 911 334305.004733 2163 100 0.000000 35.954435 0.000000 1 / [45827.632393] gssproxy 1188 1452.517418 1 120 0.000000 0.032283 0.000000 1 / [45827.644819] munged 1830 2320.307160 1 120 0.000000 0.071973 0.000000 1 / [45827.657262] kworker/26:2 1822 353897.136145 43897 120 0.000000 898.239105 0.000000 1 / [45827.669705] ptlrpc_hr01_000 3460 336124.318137 442904 120 0.000000 17810.365992 0.000000 1 / [45827.682193] osp-syn-8-0 3619 336184.665862 220211 120 0.000000 5882.553207 0.000000 1 / [45827.694667] mdt01_005 3770 353872.723671 1808687 120 0.000000 173290.311914 0.000000 1 / [45827.707132] ldlm_cn01_005 3793 353872.671213 300584 120 0.000000 17277.107604 0.000000 1 / [45827.719582] mdt_rdpg01_004 3847 334532.939989 245310 120 0.000000 17687.569537 0.000000 1 / [45827.732064] mdt_io01_005 9896 334591.192550 46646 120 0.000000 3763.810179 0.000000 1 / [45827.744524] ldlm_cn01_031 34308 296078.003738 85479 120 0.000000 4496.821664 0.000000 1 / [45827.756995] kworker/26:0 77562 351670.358704 3 120 0.000000 0.116027 0.000000 1 / [45827.771171] cpu#27, 2693.251 MHz [45827.774794] .nr_running : 0 [45827.779969] .load : 0 [45827.785137] .nr_switches : 27950216 [45827.790991] .nr_load_updates : 5335200 [45827.796748] .nr_uninterruptible : 1802 [45827.802203] .next_balance : 4340.492841 [45827.808346] .curr->pid : 0 [45827.813527] .clock : 45827813.144777 [45827.820042] .cpu_load[0] : 0 [45827.825216] .cpu_load[1] : 0 [45827.830383] .cpu_load[2] : 0 [45827.835548] .cpu_load[3] : 0 [45827.840713] .cpu_load[4] : 0 [45827.845877] .avg_idle : 1000000 [45827.851634] .max_idle_balance_cost : 500000 [45827.858972] cfs_rq[27]:/ [45827.861824] .exec_clock : 0.000000 [45827.867684] .MIN_vruntime : 0.000001 [45827.873535] .min_vruntime : 287043.884306 [45827.879864] .max_vruntime : 0.000001 [45827.885709] .spread : 0.000000 [45827.891554] .spread0 : 6468.886781 [45827.897703] .nr_spread_over : 0 [45827.902882] .nr_running : 0 [45827.908054] .load : 0 [45827.913220] .runnable_load_avg : 0 [45827.918378] .blocked_load_avg : 0 [45827.923553] .tg_load_avg : 0 [45827.928733] .tg_load_contrib : 0 [45827.933915] .tg_runnable_contrib : 0 [45827.939079] .tg->runnable_avg : 0 [45827.944257] .tg->cfs_bandwidth.timer_active: 0 [45827.949423] .throttled : 0 [45827.954597] .throttle_count : 0 [45827.959775] .avg->runnable_avg_sum : 0 [45827.964955] .avg->runnable_avg_period : 47043 [45827.972198] rt_rq[27]:/ [45827.974948] .rt_nr_running : 0 [45827.980123] .rt_throttled : 0 [45827.985288] .rt_time : 0.000000 [45827.991142] .rt_runtime : 950.000000 [45827.998901] runnable tasks: [45828.002037] task PID tree-key switches prio wait-time sum-exec sum-sleep [45828.014083] ---------------------------------------------------------------------------------------------------------- [45828.026050] watchdog/27 144 -11.915054 11459 0 0.000000 189.454936 0.000000 1 / [45828.038500] migration/27 145 0.000000 1063 0 0.000000 1100.156265 0.000000 1 / [45828.050959] ksoftirqd/27 146 286521.483176 4579 120 0.000000 113.661644 0.000000 1 / [45828.063410] kworker/27:0H 148 729.577442 9 100 0.000000 0.068988 0.000000 1 / [45828.075895] kworker/27:1 249 287031.925048 43194 120 0.000000 913.232453 0.000000 1 / [45828.088359] ttm_swap 506 179.657113 2 100 0.000000 0.058052 0.000000 1 / [45828.100827] kworker/27:1H 1036 281495.343111 4936 100 0.000000 78.079706 0.000000 1 / [45828.113271] munged 1831 1286.933827 1 120 0.000000 0.037411 0.000000 1 / [45828.125733] atd 1929 285912.101703 16 120 0.000000 5.180735 0.000000 1 / [45828.138189] kworker/27:2 1989 1324.986731 2 120 0.000000 0.009940 0.000000 1 / [45828.150682] ldlm_cn01_001 3546 277006.083452 383236 120 0.000000 22715.576986 0.000000 1 / [45828.163154] ldlm_cn01_002 3547 277004.536201 484636 120 0.000000 29861.361046 0.000000 1 / [45828.175624] mdt01_004 3769 287024.514722 1394824 120 0.000000 130062.217139 0.000000 1 / [45828.188091] ldlm_cn01_008 3800 284227.997535 359429 120 0.000000 21127.577639 0.000000 1 / [45828.200573] mdt_out01_016 7993 286257.108341 366024 120 0.000000 24481.618723 0.000000 1 / [45828.214760] cpu#28, 2693.251 MHz [45828.218383] .nr_running : 0 [45828.223558] .load : 0 [45828.228739] .nr_switches : 28258379 [45828.234592] .nr_load_updates : 5377756 [45828.240341] .nr_uninterruptible : 1963 [45828.245813] .next_balance : 4340.493285 [45828.251955] .curr->pid : 0 [45828.257128] .clock : 45828256.174033 [45828.263659] .cpu_load[0] : 0 [45828.268834] .cpu_load[1] : 0 [45828.274001] .cpu_load[2] : 0 [45828.279162] .cpu_load[3] : 0 [45828.284322] .cpu_load[4] : 0 [45828.289502] .avg_idle : 1000000 [45828.295275] .max_idle_balance_cost : 500000 [45828.302614] cfs_rq[28]:/ [45828.305454] .exec_clock : 0.000000 [45828.311315] .MIN_vruntime : 0.000001 [45828.317176] .min_vruntime : 292046.254334 [45828.323527] .max_vruntime : 0.000001 [45828.329374] .spread : 0.000000 [45828.335233] .spread0 : 11461.529963 [45828.341473] .nr_spread_over : 0 [45828.346639] .nr_running : 0 [45828.351811] .load : 0 [45828.356983] .runnable_load_avg : 0 [45828.362160] .blocked_load_avg : 0 [45828.367333] .tg_load_avg : 0 [45828.372508] .tg_load_contrib : 0 [45828.377673] .tg_runnable_contrib : 0 [45828.382854] .tg->runnable_avg : 0 [45828.388035] .tg->cfs_bandwidth.timer_active: 0 [45828.393214] .throttled : 0 [45828.398404] .throttle_count : 0 [45828.403580] .avg->runnable_avg_sum : 0 [45828.408741] .avg->runnable_avg_period : 46332 [45828.415991] rt_rq[28]:/ [45828.418742] .rt_nr_running : 0 [45828.423908] .rt_throttled : 0 [45828.429078] .rt_time : 0.000000 [45828.434932] .rt_runtime : 950.000000 [45828.442670] runnable tasks: [45828.445820] task PID tree-key switches prio wait-time sum-exec sum-sleep [45828.457869] ---------------------------------------------------------------------------------------------------------- [45828.469844] watchdog/28 149 -11.914376 11459 0 0.000000 199.092903 0.000000 1 / [45828.482289] migration/28 150 0.000000 1610 0 0.000000 1126.793754 0.000000 1 / [45828.494729] ksoftirqd/28 151 284919.094281 3348 120 0.000000 80.200607 0.000000 1 / [45828.507188] kworker/28:0 152 -2.435961 21 120 0.000000 0.110295 0.000000 1 / [45828.519664] kworker/28:0H 153 590.102824 9 100 0.000000 0.054489 0.000000 1 / [45828.532121] kworker/28:1 247 292034.299666 42823 120 0.000000 954.147415 0.000000 1 / [45828.544561] kworker/28:1H 906 291177.158747 2778 100 0.000000 65.160985 0.000000 1 / [45828.557039] mcelog 1141 817.541105 1 120 0.000000 1.428761 0.000000 1 / [45828.569508] ll_mgs_0002 3554 292044.997837 34750 120 0.000000 2385.998006 0.000000 1 / [45828.581974] mdt_rdpg01_010 4800 286311.007314 307351 120 0.000000 23110.599588 0.000000 1 / [45828.594447] ldlm_bl_07 7995 291446.886368 194076 120 0.000000 6963.209824 0.000000 1 / [45828.606915] mdt01_021 22107 280376.207664 670797 120 0.000000 61277.345754 0.000000 1 / [45828.619354] ldlm_cn01_030 34267 292034.505010 161391 120 0.000000 9055.369406 0.000000 1 / [45828.631829] mdt01_022 35178 286999.269461 202907 120 0.000000 18999.769345 0.000000 1 / [45828.645987] cpu#29, 2693.251 MHz [45828.649609] .nr_running : 0 [45828.654782] .load : 0 [45828.659963] .nr_switches : 28038357 [45828.665808] .nr_load_updates : 5342716 [45828.671564] .nr_uninterruptible : 1982 [45828.677029] .next_balance : 4340.493716 [45828.683166] .curr->pid : 0 [45828.688335] .clock : 45828688.187997 [45828.694883] .cpu_load[0] : 0 [45828.700055] .cpu_load[1] : 0 [45828.705225] .cpu_load[2] : 0 [45828.710397] .cpu_load[3] : 0 [45828.715555] .cpu_load[4] : 0 [45828.720728] .avg_idle : 1000000 [45828.726472] .max_idle_balance_cost : 500000 [45828.733803] cfs_rq[29]:/ [45828.736642] .exec_clock : 0.000000 [45828.742487] .MIN_vruntime : 0.000001 [45828.748340] .min_vruntime : 283599.454955 [45828.754676] .max_vruntime : 0.000001 [45828.760521] .spread : 0.000000 [45828.766372] .spread0 : 3004.908704 [45828.772507] .nr_spread_over : 0 [45828.777686] .nr_running : 0 [45828.782860] .load : 0 [45828.788033] .runnable_load_avg : 0 [45828.793219] .blocked_load_avg : 0 [45828.798392] .tg_load_avg : 0 [45828.803563] .tg_load_contrib : 0 [45828.808731] .tg_runnable_contrib : 0 [45828.813897] .tg->runnable_avg : 0 [45828.819057] .tg->cfs_bandwidth.timer_active: 0 [45828.824233] .throttled : 0 [45828.829412] .throttle_count : 0 [45828.834592] .avg->runnable_avg_sum : 0 [45828.839757] .avg->runnable_avg_period : 47078 [45828.847004] rt_rq[29]:/ [45828.849752] .rt_nr_running : 0 [45828.854908] .rt_throttled : 0 [45828.860084] .rt_time : 0.000000 [45828.865936] .rt_runtime : 950.000000 [45828.873650] runnable tasks: [45828.876789] task PID tree-key switches prio wait-time sum-exec sum-sleep [45828.888865] ---------------------------------------------------------------------------------------------------------- [45828.900849] watchdog/29 154 -11.913647 11459 0 0.000000 188.288680 0.000000 1 / [45828.913293] migration/29 155 0.000000 1029 0 0.000000 1096.339004 0.000000 1 / [45828.925766] ksoftirqd/29 156 278398.432441 2766 120 0.000000 69.553887 0.000000 1 / [45828.938237] kworker/29:0H 158 1593.383720 9 100 0.000000 0.058374 0.000000 1 / [45828.950694] kworker/29:1 248 283587.500718 42271 120 0.000000 796.521059 0.000000 1 / [45828.963221] ldlm_cn01_000 3545 283587.698039 435686 120 0.000000 26356.067083 0.000000 1 / [45828.975688] mdt01_007 3778 280929.438234 1275059 120 0.000000 120592.716977 0.000000 1 / [45828.988157] kworker/29:1H 3782 277141.665592 2605 100 0.000000 59.965451 0.000000 1 / [45829.000643] mdt_out01_005 4805 279352.769213 912733 120 0.000000 59765.719751 0.000000 1 / [45829.013124] ldlm_cb01_003 8780 281941.817259 368546 120 0.000000 21777.541109 0.000000 1 / [45829.025597] kworker/29:2 12323 47377.453656 2 120 0.000000 0.048816 0.000000 1 / [45829.038072] mdt_io01_007 15900 277058.808729 4020 120 0.000000 349.721224 0.000000 1 / [45829.050520] mdt_rdpg01_013 32883 278896.707359 134336 120 0.000000 9496.857252 0.000000 1 / [45829.062957] ldlm_cn01_022 34144 278110.529269 135989 120 0.000000 7813.420487 0.000000 1 / [45829.075451] ldlm_cn01_027 34150 280906.748492 129134 120 0.000000 7288.616689 0.000000 1 / [45829.087903] ldlm_bl_18 36871 283149.150628 48540 120 0.000000 1310.446366 0.000000 1 / [45829.102041] cpu#30, 2693.251 MHz [45829.105667] .nr_running : 0 [45829.110855] .load : 0 [45829.116031] .nr_switches : 29774225 [45829.121883] .nr_load_updates : 5480498 [45829.127637] .nr_uninterruptible : 1891 [45829.133093] .next_balance : 4340.494172 [45829.139241] .curr->pid : 0 [45829.144415] .clock : 45829144.209027 [45829.150954] .cpu_load[0] : 0 [45829.156125] .cpu_load[1] : 0 [45829.161304] .cpu_load[2] : 0 [45829.166471] .cpu_load[3] : 0 [45829.171645] .cpu_load[4] : 0 [45829.176810] .avg_idle : 1000000 [45829.182570] .max_idle_balance_cost : 500000 [45829.189933] cfs_rq[30]:/ [45829.192773] .exec_clock : 0.000000 [45829.198625] .MIN_vruntime : 0.000001 [45829.204478] .min_vruntime : 289553.782802 [45829.210825] .max_vruntime : 0.000001 [45829.216678] .spread : 0.000000 [45829.222529] .spread0 : 8948.980632 [45829.228666] .nr_spread_over : 0 [45829.233832] .nr_running : 0 [45829.239001] .load : 0 [45829.244158] .runnable_load_avg : 0 [45829.249325] .blocked_load_avg : 0 [45829.254491] .tg_load_avg : 0 [45829.259663] .tg_load_contrib : 0 [45829.264828] .tg_runnable_contrib : 0 [45829.270008] .tg->runnable_avg : 0 [45829.275192] .tg->cfs_bandwidth.timer_active: 0 [45829.280375] .throttled : 0 [45829.285556] .throttle_count : 0 [45829.290723] .avg->runnable_avg_sum : 0 [45829.295897] .avg->runnable_avg_period : 47758 [45829.303145] rt_rq[30]:/ [45829.305903] .rt_nr_running : 0 [45829.311076] .rt_throttled : 0 [45829.316272] .rt_time : 0.000000 [45829.322123] .rt_runtime : 950.000000 [45829.329843] runnable tasks: [45829.332972] task PID tree-key switches prio wait-time sum-exec sum-sleep [45829.345028] ---------------------------------------------------------------------------------------------------------- [45829.357010] watchdog/30 159 -11.914877 11459 0 0.000000 182.323048 0.000000 1 / [45829.369457] migration/30 160 0.000000 958 0 0.000000 1094.835190 0.000000 1 / [45829.381906] ksoftirqd/30 161 284972.856437 2709 120 0.000000 64.757549 0.000000 1 / [45829.394364] kworker/30:0H 163 584.283570 9 100 0.000000 0.044931 0.000000 1 / [45829.406852] kworker/30:1 250 289529.566706 42835 120 0.000000 1049.000525 0.000000 1 / [45829.419332] kworker/30:1H 905 286757.252228 2459 100 0.000000 61.161374 0.000000 1 / [45829.431787] kworker/30:2 1528 954.242314 2 120 0.000000 0.007065 0.000000 1 / [45829.444272] ptlrpc_hr01_005 3465 286144.781123 442886 120 0.000000 17885.734809 0.000000 1 / [45829.456761] ptlrpcd_01_15 3522 289541.901522 515216 120 0.000000 17616.201083 0.000000 1 / [45829.469215] mgs_params_noti 3551 1736.504750 2 118 0.000000 0.086408 0.000000 1 / [45829.481713] ldlm_cn01_003 3783 286006.954208 414378 120 0.000000 25366.863296 0.000000 1 / [45829.494184] ldlm_cn01_009 3802 280212.136522 346534 120 0.000000 20855.510880 0.000000 1 / [45829.506661] ldlm_cn01_010 3843 289529.876691 417464 120 0.000000 25486.105226 0.000000 1 / [45829.519134] mdt01_015 7035 287851.400014 1704974 120 0.000000 163255.578869 0.000000 1 / [45829.531589] mdt01_017 8314 288596.557440 756469 120 0.000000 72044.715802 0.000000 1 / [45829.545757] cpu#31, 2693.251 MHz [45829.549380] .nr_running : 0 [45829.554553] .load : 0 [45829.559710] .nr_switches : 28234185 [45829.565563] .nr_load_updates : 5263185 [45829.571323] .nr_uninterruptible : 1922 [45829.576780] .next_balance : 4340.494616 [45829.582930] .curr->pid : 0 [45829.588103] .clock : 45829583.006204 [45829.594642] .cpu_load[0] : 0 [45829.599807] .cpu_load[1] : 0 [45829.604983] .cpu_load[2] : 0 [45829.610149] .cpu_load[3] : 0 [45829.615328] .cpu_load[4] : 0 [45829.620497] .avg_idle : 1000000 [45829.626274] .max_idle_balance_cost : 500000 [45829.633599] cfs_rq[31]:/ [45829.636446] .exec_clock : 0.000000 [45829.642297] .MIN_vruntime : 0.000001 [45829.648156] .min_vruntime : 278539.285917 [45829.654493] .max_vruntime : 0.000001 [45829.660343] .spread : 0.000000 [45829.666197] .spread0 : -2075.088116 [45829.672431] .nr_spread_over : 0 [45829.677610] .nr_running : 0 [45829.682783] .load : 0 [45829.687944] .runnable_load_avg : 0 [45829.693118] .blocked_load_avg : 0 [45829.698291] .tg_load_avg : 0 [45829.703456] .tg_load_contrib : 0 [45829.708621] .tg_runnable_contrib : 0 [45829.713788] .tg->runnable_avg : 0 [45829.718954] .tg->cfs_bandwidth.timer_active: 0 [45829.724121] .throttled : 0 [45829.729294] .throttle_count : 0 [45829.734466] .avg->runnable_avg_sum : 0 [45829.739630] .avg->runnable_avg_period : 47030 [45829.746869] rt_rq[31]:/ [45829.749610] .rt_nr_running : 0 [45829.754785] .rt_throttled : 0 [45829.759960] .rt_time : 0.000000 [45829.765805] .rt_runtime : 950.000000 [45829.773503] runnable tasks: [45829.776633] task PID tree-key switches prio wait-time sum-exec sum-sleep [45829.788690] ---------------------------------------------------------------------------------------------------------- [45829.800643] watchdog/31 164 -11.914697 11459 0 0.000000 183.857112 0.000000 1 / [45829.813100] migration/31 165 0.000000 1094 0 0.000000 743.766734 0.000000 1 / [45829.825558] ksoftirqd/31 166 277576.959926 2799 120 0.000000 73.883662 0.000000 1 / [45829.838017] kworker/31:0H 168 907.706479 9 100 0.000000 0.044863 0.000000 1 / [45829.850483] kworker/31:1 251 278527.301060 41804 120 0.000000 904.983278 0.000000 1 / [45829.862993] kworker/31:1H 904 275101.677428 2591 100 0.000000 52.933666 0.000000 1 / [45829.875440] ptlrpcd_01_01 3508 278528.015243 513935 120 0.000000 17595.009356 0.000000 1 / [45829.887950] osp-pre-10-0 3624 278527.466430 13002 120 0.000000 654.156264 0.000000 1 / [45829.900407] ldlm_cn01_017 7590 274324.630809 242400 120 0.000000 13512.098460 0.000000 1 / [45829.912861] mdt_out01_014 7673 275439.353994 372374 120 0.000000 23637.688197 0.000000 1 / [45829.925349] mdt01_020 22106 273828.551747 610284 120 0.000000 56797.863604 0.000000 1 / [45829.937809] kworker/31:0 56178 233663.360702 3 120 0.000000 0.008434 0.000000 1 / [45829.952218] INFO: rcu_sched detected stalls on CPUs/tasks: { 19} (detected by 19, t=6 jiffies, g=4335908, c=4335907, q=74207) [45829.964923] Task dump for CPU 19: [45829.968636] bash R running task 0 78778 78771 0x00000080 [45829.976527] Call Trace: [45829.979256] [] sched_show_task+0xa8/0x110 [45829.986275] [] dump_cpu_task+0x39/0x70 [45829.992305] [] rcu_dump_cpu_stacks+0x90/0xd0 [45829.998917] [] rcu_check_callbacks+0x6f2/0x730 [45830.005729] [] ? tick_sched_do_timer+0x50/0x50 [45830.012525] [] update_process_times+0x46/0x80 [45830.019233] [] tick_sched_handle+0x30/0x70 [45830.025649] [] tick_sched_timer+0x39/0x80 [45830.031968] [] __hrtimer_run_queues+0x10e/0x270 [45830.038860] [] hrtimer_interrupt+0xaf/0x1d0 [45830.045375] [] local_apic_timer_interrupt+0x3b/0x60 [45830.052671] [] smp_apic_timer_interrupt+0x43/0x60 [45830.059773] [] apic_timer_interrupt+0x16a/0x170 [45858.290546] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib)